Category Archives: Notes

Using monolog with codeigniter4

The standard logger in codeigniter4 is fine but its no monolog!

To use monolog codeigniter4, add the following to your app/Config/Service.php file. You can read more about supported handlers and formatters in monolog documentation. See https://github.com/Seldaek/monolog/blob/main/doc/02-handlers-formatters-processors.md#formatters. and there are many third party handlers and formatters available.

  • Note that I am logging to syslog rather than to a file (this won’t work on Windows). I think logging to syslog is better if you already have a scrapper running. And also, correlating with web-server logs is bit easier. Perhaps I am mistaken.
  • I am using a third party formatter http://github.com/bramus/monolog-colored-line-formatter to add colors to console logger. I think colors are a good idea if you need to scroll and look at the logs while developing.
<?php

// app/Config/Services.php 

use Bramus\Monolog\Formatter\ColoredLineFormatter;
use Monolog\Formatter\LineFormatter;
use Monolog\Handler\BrowserConsoleHandler;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\SyslogHandler;
use Monolog\Logger;

class Services Extends BaseService 
{
    // Other services are not shown.

    /**
     * Use monolog logger.
     *
     * - Logs to syslogs
     * - Logs to console (colored)
     * - Logs to browser console (development only).
     */
    public static function logger(bool $getShared = true): Logger
    {
        if ($getShared) {
            return static::getSharedInstance('logger');
        }

        $logger = new Logger('my-portal');
        $consoleHandler = new StreamHandler('php://stdout', \Monolog\Level::Info);
        $consoleHandler->setFormatter(new ColoredLineFormatter());
        $logger->pushHandler($consoleHandler);

        // Also log to syslog
        $facilityName = "local6"; // See list here https://en.wikipedia.org/wiki/Syslog#Facility_levels
        $sysLogHandler = new SyslogHandler('my-portal', $facilityName);
        $formatter = new LineFormatter("%channel%.%level_name%: %message% %extra%");
        $sysLogHandler->setFormatter($formatter);
        $logger->pushHandler($sysLogHandler);

        if(ENVIRONMENT === "development") {
            $logger->pushHandler(new BrowserConsoleHandler());
        }

        return $logger;
    }
}

Continue to use your log_message function as before, and perhaps comment out all but one handlers in app/Config/Logger.php file.

I think once service with alias logger is added, most handlers in file app/Config/Logger.php stops working by themselves.

Reading ntfy stream in vue3-ts page using axios

The following worked for me


onMounted(async () => { refreshMetricInterval = setInterval((_) => { fetchData() }, C.DEFAULT_POLL_INTERVAL_SEC * 1000) // Subscribe to ntfy. const ntfy = await axios.get('https://ntfy.sh/mytopic/json', { headers: { Authorization: 'Bearer yout_token', Accept: 'text/event-stream', }, responseType: 'stream', adapter: 'fetch', }); // This doesn't work. We are using fetch API and the below solutions work.s // // ``` // console.debug('axa1', ntfy.data) // ntfy.data.on('message', (data) => { // console.debug(444, data); // }); // ``` for await (const data of ntfy.data) { const json = JSON.parse(new TextDecoder().decode(data)); console.debug('Got line from ntfy', json); } })

Changing client generated by openapi-generated-cli tool by modifying templates

I needed to add more derive traits on the enum generated by openapi-generate-cli e.g. strum::EnumIter.

  • First, dump the templates using the following command openapi-generator-cli author template -g rust . This will write templates to the directory out.
  • Now edit the templates inside out directory. I had to add strum to Cargo.mutache and strum::EnumIter inside derive of all pub enum inside model.mustache file.
  • Now regenerate the client using openapi-generator-cli generate -t out -i openapi.json -g rust -o api-client-rs

And you have updated client.

Here are some diffs

86d3b88c268545798a7cf5a917a99052 -- Screenshot of code

PS: I could not get the local version to run on my system. I used the following docker command

    podman run --rm \
        -v $PWD:/local openapitools/openapi-generator-cli generate \
        -i https://beta.oneapi.dognosis.link/api/v1/openapi.json \
        -t /local/templates \
        -g rust \
        -o /local/oneapi-client-rs

A simple `leptos` app

rust-projects/leptos at main · dilawar/rust-projects

3691284a54cc44739171c1cf66b10e3c -- A simple app using Rust + Leptos. A form that stores data to LocalStorage. Its reactive!

cbeb303e4778433c9071bdec36216919 -- Audio recording in a Rust + Leptos app

There is also a QR scanner that may or may not work in your browser. I used a third party crate that I patched to compile with Leptos 0.7 but did not test is thoroughly.

leptos-use (https://leptos-use.rs/) makes it easy to access browser’s API such a audio/video streams. The audio recorder records the audio in chunk of a few seconds and plot the raw values on the canvas. The plotting is very basic here!

I tried using https://thawui.vercel.app/ components but could not make it work with my slightly complicated hooks that update the local storage. I reverted back to standard leptos input component.

## [component]
pub fn Form() -> impl IntoView {
    let storage_key = RwSignal::new("".to_string());

    let (state, set_state, _) = use_local_storage::<KeyVal, JsonSerdeCodec>(storage_key);

    let upload_patient_consent_form = move |file_list: FileList| {
        let len = file_list.length();
        for i in 0..len {
            if let Some(file) = file_list.get(i) {
                tracing::info!("File to upload: {}", file.name());
            }
        }
    };

    view! {
        <h5>"Form"</h5>
        <Space vertical=true class=styles::ehr_list>

            // Everything starts with this key
            <ListItem label="Code".to_string()>
                <input bind:value=storage_key />
            </ListItem>

            // Patient
            <InputWithLabel key="phone".to_string() state set_state></InputWithLabel>
            <InputWithLabel key="name".to_string() state set_state></InputWithLabel>
            <SelectWithLabel
                key="gender".to_string()
                options=Gender::iter().map(|x| x.to_string()).collect()
                state
                set_state
            ></SelectWithLabel>
            <InputWithLabel key="extra".to_string() state set_state></InputWithLabel>

        </Space>
    }
}

## [derive(Debug, strum::EnumIter, strum::Display)]
enum Gender {
    Male,
    Female,
    Other,
}

## [component]
pub fn InputWithLabel(
    key: String,
    state: Signal<KeyVal>,
    set_state: WriteSignal<KeyVal>
) -> impl IntoView {
    let label = key.split("_").join(" ");
    let key1 = key.to_string();

    view! {
        <Flex>
            <Label>{label}</Label>
            // Could not get thaw::Input to change when value in parent changes.
            <input
                prop:value=move || {
                    state.get().0.get(&key1).map(|x| x.to_string()).unwrap_or_default()
                }
                on:input=move |e| {
                    set_state
                        .update(|s| {
                            s.0.insert(key.to_string(), event_target_value(&e));
                        })
                }
            />
        </Flex>
    }
}

// SelectWithLabel is not shown. See the linked repo for updated code.

vue: navigator.mediaDevices is undefined

If you are using internal IPs e.g. 192.168.0.300 etc for development, you are likely to encounter this error when dealing with media devices.

Assuming that you are not using localhost for development for a reason, you can do either of the following

  1. In Firefox you can enable the following two options described in https://stackoverflow.com/a/66605018/1805129. Go to about:config
    set to true media.devices.insecure.enabled and media.getusermedia.insecure.enabled

90c8821922ca484ba9ad755c39c9adbe -- <code>about:config</code> options in firefox” title=”” class=”aligncenter” /></a></p>
<ol>
<li>You can use service like <code>ngrok</code> or <a href=https://serveo.net/ to temporarily get https address. I learnt about these two services in the README file of https://www.npmjs.com/package/vue-qrcode-reader.

self-hosted ntfy service with admin account

  • First, we need a script that setup admin account for us after starting the service. init.sh file that you should put inside ntfy/init.sh
    #!/bin/sh
    ntfy serve &
    sleep 2
    NTFY_PASSWORD=${NTFY_PASSWORD} ntfy user add --role=admin ${NTFY_USER}
    wait
    
  • Now the compose file compose.yaml or docker-compose.yaml file.
    services:
      ntfy:
        image: binwiederhier/ntfy
        container_name: ntfy
        entrypoint: [ "/bin/sh", "/var/lib/ntfy/init.sh" ]
        environment:
          - TZ=UTC    \# optional: set desired timezone
          - NTFY_BASE_URL=https://ntfy.dilawars.me
          - NTFY_AUTH_FILE=/var/lib/ntfy/auth.db
          - NTFY_CACHE_FILE=/var/lib/ntfy/cache.db
          - NTFY_AUTH_DEFAULT_ACCESS=deny-all
          - NTFY_ATTACHMENT_CACHE_DIR=/var/lib/ntfy/attachments
          - NTFY_ENABLE_LOGIN=true
          - NTFY_USER=admin
          - NTFY_PASSWORD=yoyodillusingh
        volumes:
          - /var/cache/ntfy:/var/cache/ntfy
          - /etc/ntfy:/etc/ntfy
          - ./ntfy/:/var/lib/ntfy
        ports:
          - 8001:80
        healthcheck: \# optional: remember to adapt the host:port to your environment
            test: ["CMD-SHELL", "wget -q --tries=1 http://localhost:80/v1/health -O - | grep -Eo '\"healthy\"\\s*:\\s*true' || exit 1"]
            interval: 60s
            timeout: 10s
            retries: 3
            start_period: 40s
        restart: unless-stopped
    

Composition of coconut water

Composition of coconut water from nuts at 6 and 12 months maturity

Components 6 Months maturity 12 Months maturity
Water 94.18 94.45
Total solids 5.82 5.55
Protein 0.12 0.52
Fat 0.07 0.15
Ash 0.87 0.47
Carbohydrates 4.76 4.41
Dietary fiber ND ND
Total sugars 5.23 3.42
Sucrose 0.06 0.51
Glucose 2.61 1.48
Fructose 2.65 1.43
Potassium mg% 203.70 251.52
Calcium mg% 27.35 31.64
Sodium mg% 1.75 16.10
Phosphorus mg% 4.66 12.79
Magnesium mg% 6.40 9.44

Source: Yong, J.W., Ge, L., Ng, Y.F., Tan, S.N., 2009. The chemical composition and biological properties of coconut (Cocos nucifera L.) water. Molecules 14 (12), 5144–5164.

psql: FATAL: role “root” does not exist.

I was using healthcheck in postgres container which triggered this ominous error.

I made the following changed and the problem went away (for obvious reason, I guess) — pg_isready needs to know which user is being problem.

Following is the diff.

index 4c91dcc..89ad702 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -6,8 +6,9 @@ services:
     shm_size: 512mb
     environment:
       POSTGRES_PASSWORD: test_postgres_123
+      POSTGRES_USER: test_postgres_user
     healthcheck:
-      test: ['CMD-SHELL', 'pg_isready' ]
+      test: "pg_isready -U $$POSTGRES_USER"
       interval: 1s
       timeout: 5s
       retries: 10

Thanks https://stackoverflow.com/questions/60193781/postgres-with-docker-compose-gives-fatal-role-root-does-not-exist-error

How to use slot=”foobar” in Vue3

Following is Vue2 code which is refactored for Vue3 below.

<f7-list-input label="Search patient" :input="false">
    <vue-simple-suggest
        slot="input"
      class="item-input-wrap"
      :max-suggestions="10"
      :nullable-select="false"
      :min-length="4"
      @update:model-select="onPatientSelected"
      :list="queryPatient"
    >
    </vue-simple-suggest>
</f7-list-input>
<f7-list-input label="Search patient" :input="false">
  <template v-slot:input>
    <vue-simple-suggest
      class="item-input-wrap"
      :max-suggestions="10"
      :nullable-select="false"
      :min-length="4"
      @update:model-select="onPatientSelected"
      :list="queryPatient"
    >
    </vue-simple-suggest>
  </template>
</f7-list-input>

Querying a directory of JSONs using duckdb

To make this problem concrete, I’ve downloaded database of all CVEs from https://github.com/CVEProject/cvelistV5 using git clone. I want to create API that can query this repository. I can think of the following available options.

  1. grep based tool e.g. git grep , rg , grep etc.
  2. https://jqlang.github.io/jq/ like tools.
  3. Somehow ingest JSONs into a relational database and use SQL either using a third party tool or using a custom solution.

I’d have preferred 1 or 2 if I was writing a cli application. Since I am building a RESTful API, I’d prefer 3 since queries will be easier to write and integrate into other applications. Custom solution is also not a bad idea if existing tooling is not good enough or there are other constraints. Most databases like PostgreSQL and sqlite3 allows JSON to be inserted and queried.

While searching for such plugins, I came across https://duckdb.org/ which seems to support this use cases. See for example https://duckdb.org/docs/data/multiple_files/overview 💯.

For this exercise, I am not interested in performance. I can use caching to improve performance drastically later is need arise. I am looking for a solution that has the best DX — and if I am lucky doesn’t easily allow stupid mistakes!

Installation

The installation was a breeze. Single binary! It can also be used as Rust/Python library and third party integration for PHP are also available 💯.

$ wget https://github.com/duckdb/duckdb/releases/download/v1.1.3/duckdb_cli-linux-amd64.zip

$ unzip duckdb_cli-linux-amd64.zip 
Archive:  duckdb_cli-linux-amd64.zip
  inflating: duckdb                  
(PY311) [dilawar@rasmalai cve_database_git (master)]$ ./duckdb 
v1.1.3 19864453f7
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D 

Great, seems to work!

Querying JSON files

For a sanity check, I did a “loopback” — print what you read. Just to make sure that engine is parsing JSON files. I passed a glob that matches all JSON files. Loading all files together raised an error — mismatch in schema.

D SELECT * FROM 'cves/**/*.json';
Invalid Input Error: JSON transform error in file "cves/2001/1xxx/CVE-2001-1517.json", in record/value 1: Object {"affected":[{"product":"n/a","vendor":"n/a","vers... has unknown key "tags"
Try increasing 'sample_size', reducing 'maximum_depth', specifying 'columns', 'format' or 'records' manually, setting 'ignore_errors' to true, or setting 'union_by_name' to true when reading multiple files with a different structure.
D 

Fair enough warning about inconsistent schema across files and it suggested what I should explore. I like tools that hints at what to do in case of error💯.

I am not sure which one is the best option here though: ignore_errors or perhaps union_by_name is a better idea🤔? I am going ahead with union_by_name. I had to tweak the query little bit: SELECT * FROM read_json('cves/2000/**/*.json', union_by_name = true);

image.png

Nice!

Note that we are not yet inserting these JSON record into to a SQL table but rather processing them on the fly. I say this because I don’t see the opened database test.duck.db size to increase at all!

To insert these JSON records into SQL table that can be queries later, we use the following

D CREATE TABLE cve AS SELECT * FROM read_json_auto('./cves/2003/*/*.json', union_by_name = true);
100% ▕████████████████████████████████████████████████████████████▏ 
D select COUNT(*) FROM cve;
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│         1553 │
└──────────────┘

I am ready to append to this table and write queries.

Using python module

I did the same exercise as before using duckdb python module.

 import duckdb                                                                                                                               

 git_repo = _cve_repo_dir()                                                                                                                  
 logger.info(f"Syching duckdb from {git_repo}...")                                                                                           
 json_files = list(git_repo.glob("cves/*/**/*.json"))                                                                                        

 con = duckdb.connect("cve.duck.db")                                                                                                         

 \# create table if it doesn't exists using a JSON.                                                                                           
 first_json = json_files[-1]                                                                                                                 
 logger.info(f"> Creating table using {first_json}")                                                                                         
 con.sql(                                                                                                                                    
    f"CREATE TABLE IF NOT EXISTS cves AS SELECT * FROM read_json('{str(first_json)}', union_by_name = True)"                                
 )                                                                                                                                           
 s = con.sql("SELECT * FROM cves;")                                                                                                          
 print(s)                                                                                                                                    
2024-11-30 11:46:24.723 | INFO     | utils:sync_duckdb:40 - > Creating table using /home/dilawar/Work/SUBCOM/keeda/keeda-py/cve_json_db.git/cves/2024/9xxx/CVE-2024-9999.json
┌────────────_─────────────_──────────────────────_────────────────────────────────────────────────────────────────────────────────────────────┐
│  dataType  │ dataVersion │     cveMetadata      │                                         containers                                         │
│  varchar   │   varchar   │ struct(cveid varch_  │ struct(cna struct(affected struct(defaultstatus varchar, platforms varchar[], product va_  │
├────────────┼─────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ CVE_RECORD │ 5.1         │ {'cveId': CVE-2024_  │ {'cna': {'affected': [{'defaultStatus': unaffected, 'platforms': [Windows], 'product': W_  │
└────────────┴─────────────┴──────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────┘

Great, once I have table initialized with schema, I can insert entries from other files: INSERT INTO cves SELECT * FROM read_json('{str(file)}'). One has to ensure that other files have same schema. If not, then we get error while inserting.

TypeMismatchException: Mismatch Type Error: Type STRUCT(cveId VARCHAR, assignerOrgId UUID, state VARCHAR, dateReserved VARCHAR, dateUpdated 
VARCHAR, dateRejected VARCHAR, assignerShortName VARCHAR) does not match with STRUCT(cveId VARCHAR, assignerOrgId UUID, state VARCHAR, 
assignerShortName VARCHAR, dateReserved VARCHAR, datePublished VARCHAR, dateUpdated VARCHAR). Cannot cast STRUCTs - element "dateRejected" in 
source struct was not found in target struct

In this case, the other files have some fields added over time and it make sense to add those columns/fields to the schema on the fly (ref: https://duckdb.org/docs/sql/statements/update.html#update-from-other-table).

But I just couldn’t figure out how to the resolve the following error. Looks like updating an existing table to support the schema of a new file is not supported out of the box.

TypeMismatchException: Mismatch Type Error: Type STRUCT(defaultStatus VARCHAR, platforms VARCHAR[], product VARCHAR, vendor VARCHAR, versions 
STRUCT(lessThan VARCHAR, status VARCHAR, "version" VARCHAR, versionType VARCHAR)[]) does not match with STRUCT(collectionURL VARCHAR, 
defaultStatus VARCHAR, packageName VARCHAR, product VARCHAR, versions STRUCT(lessThanOrEqual VARCHAR, status VARCHAR, "version" VARCHAR, 
versionType VARCHAR)[], vendor VARCHAR). Cannot cast STRUCTs of different size

The solution to this problem in this case was simple. Each sub-directory contains files with the same schema. I created a table for each sub-directory!

2.7GB worth of CVEs were stored in 456MB of database file.

[dilawar@khaja keeda-py (download_cve_database)]$ du -sh cve_json_db.git/cves/
2.7G    cve_json_db.git/cves/
[dilawar@khaja keeda-py (download_cve_database)]$ ls -ltrh cve.duck.db 
-rw-r--r-- 1 dilawar dilawar 456M Nov 30 20:56 cve.duck.db
[dilawar@khaja keeda-py (download_cve_database)]$

Querying

Now the querying part. I want to query CVEs that affect a product called wpzoom. Thanks https://github.com/duckdb/duckdb/issues/9901#issuecomment-1843178389

D select * from cve_in_year_2024_9xxx where len(list_filter(containers->>'$..affected[*].vendor', x-> x == 'wpzoom'));
┌────────────_─────────────_──────────────────────_────────────────────────────────────────────────────────────────────────────────────┐
│  dataType  │ dataVersion │     cveMetadata      │                                     containers                                     │
│  varchar   │   varchar   │ struct(cveid varch_  │ struct(cna struct(providermetadata struct(orgid uuid, shortname varchar, dateupd_  │
├────────────┼─────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ CVE_RECORD │ 5.1         │ {'cveId': CVE-2024_  │ {'cna': {'providerMetadata': {'orgId': b15e7b5b-3da4-40ae-a43c-f7aa60e62599, 'sh_  │
└────────────┴─────────────┴──────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘
D 

Notes

  1. https://www.reddit.com/r/DuckDB/comments/1dvxcd6/importreading_large_json_file/
  2. https://github.com/duckdb/duckdb/issues/9901#issuecomment-1843178389

Using $\sout{\text{AJAX}}$ HTMX with Laravel + Blade

I’ve a list of uploaded files that I plan to render in a view. For each file, I want to call an API endpoint e.g. /api/v1/fileinfo to fetch some information and display it.

c7056b55f49a4c00aa56b6a26fe19f5e -- List of files in blade template

My blade template is following. A sample HTML page generated from it is shown above.

@extends('layouts.app')                                                                                                                 

@section('content')                                                                                                                     
    <div class="container">                                                                                                             
        <h3>Files</h3>                                                                                                                  

        @foreach ($files as $file)                                                                                                      
            <div class="card m-1 px-2 py-1">                                                                                            
                <div class="card-content">                                                                                              
                    {{ $file['display_name'] }}                                                                                         
                </div>                                                                                                                  
                <div>                                                                                                                   
                    <button class="btn btn-secondary">Show Information</button>                                                         
                </div>                                                                                                                  
            </div>                                                                                                                      
        @endforeach                                                                                                                     

    </div>                                                                                                                              
@endsection

Being old school, I don’t want to enable livewire/inertia just for this. I thought of of using AJAX which all the cool kids were using a decade ago.

Enter AJAX HTMX

Recently I read about HTMX. It is a good opportunity to play with it. The HTMX documentation (</> htmx ~ Documentation) is superb. The API looks great. AJAX can wait.

In code sample shown below, htmx-post define the endpoint that will be called with data defined in hx-vals when hx-trigger event occurs. The inner img with class htmx-indicator will get triggered and we’ll see a indicator spinning for a short while. The value received from server will be put into hx-target which has id $id. $id is generated randomly by PHP to create one-to-one mapping between hx-target and target div.

        @foreach ($files as $file)                                                                                                          
            <div class="card m-1 px-2 py-1">                                                                                                
                <!-- Generate a unique id to insert content received from POST                                                          
                    request -->                                                                                                             
                @php($id=uniqid("div_"))                                                                                                    
                <div class="card-content">                                                                                                  
                    {{ $file['display_name'] }}                                                                                             
                </div>                                                                                                                      
                <div>                                                                                                                       
                    <button class="btn btn-link"                                                                                            
                        hx-post="/api/v1/fileinfo"                                                                                          
                        hx-vals='{"path" : "{{ $file["path"] }}" }'                                                                         
                        hx-target='#{{ $id }}'                                                                                              
                        hx-trigger="mouseenter"                                                                                             
                        >                                                                                                                   
                        File Information                                                                                                    
                        <img class="bg-primary htmx-indicator" src="{{ asset('svg/90-ring.svg') }}" />                                      
                    </button>                                                                                                               
                    <div id="{{ $id }}">                                                                                                    
                    </div>                                                                                                                  
                </div>                                                                                                                      
            </div>                                                                                                                          
        @endforeach

Let’s see this in action. We barely see the spinner since the request doesn’t take much time.

00694da521964fe6b9c9edaf99485ca7 -- HTMX in Laravel Blade

Importantly, note that we render raw JSON😢. HTMX expects HTML from servers and doesn’t support JSON → HTML conversion natively. Well, this sucks for obvious reasons. Who returns HTML from APIs?!

So we have to turn JSON into HTML by ourselves. Thankfully there are community maintained plugins such as client-side-templates. See https://github.com/bigskysoftware/htmx-extensions/blob/main/src/client-side-templates/README.md. The plugin documentation is pretty clear. I am going to show the final solution.

  • The top-level div has hx-ext set that enables the extension.
  • hx-target is replaced by mustache-template.
  • 👉🏾 Our <template> is added. If you are using blade then you have to prefix mustache template with @ so that blade doesn’t touch them. The template create HTML out of JSON.

04c2c39271db4853b35822b149e30634 -- Code Snippet showing how to use htmx

Let’s try again! Great, I got most basic functionalities working.

05047d703e9b446d823bc8e4bff79096 -- Demo: HTMX with blade template

Apparently HTMX is quite powerful. You can search HN for resources https://hn.algolia.com/?q=htmx

Flatten a HashMap> to Vec in Rust

I have somethat that may look like the output of tree like command. I want it a plain list wihtout nesting for further processing. I want to convert the following on the left to the one on the right.

{
    "/opt": [
        "a",
        "b",
        "c",
    ],
    "/root": [
        "a",
        "b",
        "c",
    ],
}
[
    "/opt/a",
    "/opt/b",
    "/opt/c",
    "/root/a",
    "/root/b",
    "/root/c",
]

The following gist shows how to do it. You can play with code on the playground here https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=dfdbc1036a4bc502488598a742552fba.

Line 12 converts HashMap to a vector of vector where inner most vector is join of key with individual values. flatten converts vector of vector to a vector (akin to concat).

Gist

https://gist.github.com/rust-play/3a11b9e4d16e992e995c8aa0d26c6141

binwalk (and friends) is all you need to extract arbitrary binary files

Today, I wanted to look inside the firmware before uploading it to the router. I downloaded and unzipped it. It’s a pretty big file for a firmware. Inside the zip, there was a .bin
file. The venerable file utility told me it that it is a hex file.

“`
[dilawar@rasmalai keeda-rs (main)]$ file ax10v3-up-us-ver1-0-6-P1\[20240701-rel63845\]_nosign_2024-07-01_17.49.38.bin
../ax10v3-up-us-ver1-0-6-P1[20240701-rel63845]_nosign_2024-07-01_17.49.38.bin: data
“`

The all-mighty magika is much-much better but still not very helpful.

“`bash
$ magika ax10v3-up-us-ver1-0-6-P1\[20240701-rel63845\]_nosign_2024-07-01_17.49.38.bin
ax10v3-up-us-ver1-0-6-P1[20240701-rel63845]_nosign_2024-07-01_17.49.38.bin: ISO 9660 CD-ROM filesystem data (archive)
“`

Then, I played with hexdump but not with much success. Too much tribal knowledge and bravery is needed to use hexdump or similar tool. What would a coward do?

I searched around a bit and found exactly was I was looking for. This blog post Steak’s DocsReverse Engineering TP-Link TL-WA901ND firmware and obtainin…. explains exactly what I was looking for. Great! Just use binwalk. And binwalk has been rewritten in Rust! 🦀. Double great!

I asked it to extract the file using `binwalk -e`. It extracted two UBI Image files.

“`bash
[dilawar@rasmalai 034ff8a7811405e50d03c9fd06c29409b25243cd2b5bed35b1d0aafb2f793a26]$ binwalk -e ax10v3-up-us-ver1-0-6-P1\[20240701-rel63845\]_nosign_2024-07-01_17.49.38.bin
/home/dilawar/.keeda/data/034ff8a7811405e50d03c9fd06c29409b25243cd2b5bed35b1d0aafb2f793a26/extractions/ax10v3-up-us-ver1-0-6-P1[20240701-rel63845]_nosign_2024-07-01_17.49.38.bin
——————————————————————————————————————————————————————————–
DECIMAL HEXADECIMAL DESCRIPTION
——————————————————————————————————————————————————————————–
4825 0x12D9 UBI image, version: 1, image size: 23330816 bytes
——————————————————————————————————————————————————————————–
[+] Extraction of ubi data at offset 0x12D9 completed successfully
——————————————————————————————————————————————————————————–

Analyzed 1 file for 85 file signatures (187 magic patterns) in 120.0 milliseconds
“`

“`bash
[dilawar@rasmalai ubi_12D9.img]$ ls -la
total 21824
drwxr-xr-x 2 dilawar users 84 Nov 17 07:42 .
drwxr-xr-x 3 dilawar users 26 Nov 17 07:42 ..
-rw-r–r– 1 dilawar users 3936256 Nov 17 07:42 img-1957174073_vol-kernel.ubifs
-rw-r–r– 1 dilawar users 18411520 Nov 17 07:42 img-1957174073_vol-rootfs.ubifs
“`

Now can I extract what is inside UBI file? Sure I can. I wish there was a recursive extract option in binwalk. Use `binwalk -Mve` to recursively extract files. It’s super cool!

“`bash
[dilawar@rasmalai ubi_12D9.img]$ binwalk -e img-1957174073_vol-kernel.ubifs

/home/dilawar/.keeda/data/034ff8a7811405e50d03c9fd06c29409b25243cd2b5bed35b1d0aafb2f793a26/extractions/ax10v3-up-us-ver1-0-6-P1[20240701-rel63845]_nosign_2024-07-01_17.49.38.bin.extracted/12D9/ubifs-root/ubi_12D9.img/extractions/img-1957174073_vol-kernel.ubifs
——————————————————————————————————————————————————————————–
DECIMAL HEXADECIMAL DESCRIPTION
——————————————————————————————————————————————————————————–
0 0x0 uImage firmware image, header size: 64 bytes, data size: 3839011 bytes, compression: gzip, CPU: MIPS32,
OS: Linux, image type: OS Kernel Image, load address: 0x80010000, entry point: 0x8063F8E0, creation time:
2024-06-14 11:14:05, image name: “Linux-4.4.140”
——————————————————————————————————————————————————————————–
[+] Extraction of uimage data at offset 0x0 completed successfully
——————————————————————————————————————————————————————————–
“`

“`bash
[dilawar@rasmalai img-1957174073_vol-kernel.ubifs.extracted]$ cd 0/
[dilawar@rasmalai 0]$ ls -ltrha
total 3.7M
drwxr-xr-x 3 dilawar users 15 Nov 17 07:46 ..
-rw-r–r– 1 dilawar users 3.7M Nov 17 07:46 Linux-4.4.140.bin
drwxr-xr-x 2 dilawar users 31 Nov 17 07:46 .
“`

### Gotchas! Missing helper tools

`binwalk` depends on other tools to do its bidding e.g., `dtc` and `vmlinux-to-elf`
(vmlinux-to-elf). You must ensure these are installed. Run the following

“`bash
sudo apt install -y p7zip-full pipx
pipx install git+https://github.com/sviehb/jefferson.git
pipx install git+https://github.com/jrspruitt/ubi_reader
pipx install git+https://github.com/marin-m/vmlinux-to-elf
“`

Also install dtc. It’s not available on Debian. You have to manually install it. It is available on multiple other OSes though https://pkgs.org/search/?q=dtc&on=provides.

A python script to uninstall pkg file on MacOS

https://github.com/dilawar/Scripts/blob/master/pkg_uninstall.py

Usage

Pass pkg name and it will show you installed packages that matches the query and can be removed.

dilawar@halwa ~/Scripts (master)> sudo python3 pkg_uninstall.py clamav
0: com.cisco.ClamAV.programs
1: com.cisco.ClamAV.libraries
2: com.cisco.ClamAV.documentation
Select a package to uninstall 0            
Uninstalling com.cisco.ClamAV.programs
Forgot package 'com.cisco.ClamAV.programs' on '/'.
dilawar@halwa ~/Scripts (master)> sudo python3 pkg_uninstall.py clamav
0: com.cisco.ClamAV.libraries
1: com.cisco.ClamAV.documentation
Select a package to uninstall 0
Uninstalling com.cisco.ClamAV.libraries
Forgot package 'com.cisco.ClamAV.libraries' on '/'.

And that’s it.

A math tutor using NextJs/tRPC and OpenAI

Last week, I spend some time learning react framework. I did a small project where you can upload screenshot of your math problem and get step-by-step solution using OpenAI. The project was not my own idea.

https://github.com/dilawar/math-tutor-openai

I used NextJS, Typescript and tRPC for this project — this stack was suggested. In the past, I have used JavaScript but do not have great understanding of it. I am yet to read a decent book such a https://eloquentjavascript.net/.

React was totally new to me. Though I am familiar with underlying concepts or reacting programming since I’ve used Vue before. I have no opinion about nextjs, it is popular among React people so I just used it. I still don’t know how I feel about React. The Vue documentation is certainly better than React/NextJS.

https://trpc.io/ is certainly very interesting and I see its appeal. tRPC has the benefit of using the same language for both server and client side. At the time of writing his blog post, it didn’t have good support for uploading form-data (the form-data API is in beta which I didn’t try). Laravel + PHP8 is my go to stack for backend but I guess it is not popular with cool kids. PHP5 has a well deserved bad reputation which is no longer true for PHP8 but bad reputations are not easily lost.

The dependency management with npm is worse than Python’s ecosystem. I hope it gets better. Rust ecosystem and cargo has spoiled me so bad and sometimes I feel like why do I even bother with any other language!

Here is screenshot of the app. You’d need OpenAI key to run this app.

math-tutor-openai -- A simple typescript math tutor app

A simple tool to send desktop notification for a calendar event

First the rant!

For last 3 months, my Zoho Mail client lost an essential features: desktop notification when a calendar event is about to occur. It sends in-app notification but it is useless since I don’t pay attention to that. Calendar notifications are much more important since most of them are social contracts and one must not mixed them with mere notification. I missed many meetings or got late by a few minutes. A developer easily lose track of time when working!

I wrote to Zoho support and they told me that they are reimplementing it a brand new feature that will enable this again. Note to product manager — don’t break a working feature unless the implementation is ready. And they claimed they have enabled the desktop notification just for me. To this day, I am yet to see a desktop notification from Zoho Email client. I double, triple check the settings and after 10 years of experience with software development, I can’t figure is how to enable this settings, this tool is not for me!

So I wrote a small tool that does exactly this.

https://github.com/dilawar/ical-desktop-notification

Given calendar iCal url, it sends you desktop notification if an event is about to occur. At the time of this writing, “about to occur” means in 3 minutes. You can tweak this and perhaps send me a PR if you make it configurable from the cli.

How to find an iCal url? Most calendar providers should have it enabled in settings. Here is a screenshot from google calendar.

2de437906848412483032ae49b804943 -- How to find ical url from your google calendar

You should use the private URL if you also want to see the title of the event.

On windows, you can add this tool to Task Schedular.

I can’t believe I am using Windows — things you do to make a living!