Intersections:

1. Integration: ML models often serve as components within larger software systems, which are built using traditional software engineering techniques.

2. Testing and Validation: Both approaches use testing to validate that the code or model behaves as expected, although the specifics of the testing may differ.

3. Deployment and Maintenance: Whether you’re using a machine learning model or traditional software, both need to be deployed, monitored, and maintained.

4. Version Control: Both types of development often use version control systems like Git to manage changes and history. However, machine learning also requires versioning for data, models, hyperparameters, and results.

Differences:

1. Deterministic vs Probabilistic: Traditional software engineering is generally deterministic, following explicit rules and logic, while machine learning is probabilistic and learns patterns from data.

2. Interpretability: Traditional software is generally easier to debug and interpret, while machine learning models, especially complex ones, can be harder to understand.

3. Data Dependency: Machine learning is heavily dependent on data for training, validation, and testing, while traditional software engineering does not inherently require data for these phases.

4. Feedback Loop: Machine learning often involves a continuous feedback loop where the model learns from new data, whereas traditional software does not inherently have this feature.

5. Skill Set: Traditional software engineering and machine learning require different skill sets, although there’s growing overlap as software engineers become more data-savvy and data scientists learn more about software best practices.

   • ML-Specific Skills: Includes statistical and mathematical understanding (probability, statistics, linear algebra, calculus), data skills (data cleaning, preprocessing, feature engineering, visualisation), model training and evaluation (algorithm selection, hyperparameter tuning, evaluation metrics), specialised tools and libraries (machine learning frameworks, data manipulation libraries), and experimentation (A/B testing, model interpretability).

I hope the examples above give a better idea of why the architecture and design pattern of A.I. engineering is an emergent craft all on its own.

A Registry-based, Interface-Driven Design Pattern

In object-oriented programming, especially in languages like Java, C#, and TypeScript, an “interface” is a type definition that specifies a contract: a set of methods (and possibly properties) that a class must implement. The interface itself doesn’t contain any implementation. It just defines what methods should exist in any class that claims to implement that interface.

For A.I.-driven or LLM-based applications that require stability and the flexibility for experimentation, the use of interfaces and abstract classes can be especially beneficial. A.I. development often involves a lot of experimentation – tweaking models, changing preprocessing steps, using different data sources. If your system is designed with modularity in mind (enabled by interfaces and abstract design patterns), then swapping out components for experimentation becomes much easier, without the risk of breaking other parts of the system. Moreover, when scaling or deploying A.I. systems, having a modular design ensures stability and maintainability.

In short, the interface provides a clear contract on what to expect.

Combining registry, abstract classes and interfaces:

• You’d have interfaces for each type of functionality (e.g., file processing, chunking, embedding, storage, retrieval). This makes it clear what each component should do.

• The registry can then be used to manage and provide the appropriate implementation of these interfaces at runtime. This allows the system to dynamically adapt to different scenarios or configurations.

For example, consider processing different file types:

1. Interface-Driven Part:

   • Let’s begin by defining an interface. For example, consider a DataSource interface that has methods to ingest data and fetch metadata. This interface sets a clear contract for what every data source processor should do.

     from abc import ABC, abstractmethod
      
     class DataSource(ABC):
         @abstractmethod
         def ingest(self):
             pass
      
         @abstractmethod
         def get_metadata(self):
             pass

2. Registry Part:

   • Once the data is ingested (say, files from a Git repository), the system can look up the FILE_TYPE_PROCESSORS registry to find the appropriate processor based on each file’s extension.

     FILE_TYPE_PROCESSORS = {
         ".py": PythonFileProcessor,
         ".md": MarkdownFileProcessor,
         # ...
     }

When the system needs to ingest data, it can query this registry to find the appropriate processor based on each file’s extension.

3. Unified Workflow Inside DataSource Subclasses:

   • Within a DataSource subclass, you can bring these two steps together. Ingest the data first, and then use the appropriate file processor on each ingested file.

class GitRepoSource(DataSource):
    def ingest(self):
        files = self._clone_and_filter_files()
        processed_data = []
        
        for file_path in files:
            ext = self._get_file_extension(file_path)
            processor_class = FILE_TYPE_PROCESSORS.get(ext)
            if processor_class:
                processor = processor_class()
                processed_data.append(processor.process(file_path))
                
        return processed_data

class ConsumptionInterface(ABC):
    
    @abstractmethod
    def display(self, content):
        pass

class TextDisplay(ConsumptionInterface):
    def display(self, content):
        # Simply print the content
 
class VoiceOutput(ConsumptionInterface):
    def display(self, content):
        # Use Text-to-Speech to vocalise the content
 
class AnimatedVideo(ConsumptionInterface):
    def display(self, content):
        # Convert content into animated video

A General, Simplified Architecture for A.I. Apps End-to-End

The A.I./LLM app stack is just emerging and a16z had very helpfully structured it around the general stages of prompt construction (retrieval), prompt execution (inference) and some toolings within the ecosystem. Data flows and pipelines are the backend plumbing to this ecosystem which is another important distinction vis-à-vis traditional software engineering as well. I have simplified my general architecture and components of focus for development below:

Prioritisation for a 4-Month Developmental Roadmap

So there’s a whole lot of things that can be done within this general framework and architecture that I am interested in:

• Neural search over artworks

• Real-time, multilingual narration of generated stories

• Refactoring and writing tests for a codebase

• Chat with a knowledgebase over documents and data

• Code generation

• Social media content generation and publishing

For the purpose of prioritisation, given that reading comprehension and creative writing are 2 distinct skillsets and the former will be very valuable for the latter, my plan is to prioritise teaching LLMs to learn and start with less complex use cases such as neural search over artworks and a knowledgebase so a rough timeline might proceed as below:

• Month 1-2: Neural Search Over Artworks - For foundational data pipelines.

• Month 2-3: Chat with a Knowledgebase Over Documents and Data - To develop ingestion and retrieval methods and UI components.

• Month 3-4: Refactoring and Writing Tests for a Codebase - Utilise insights and methods developed during the chat/knowledgebase phase for smarter automated testing and refactoring.

With the above, even as we are focusing on building neural search and a knowledgebase, I’m thinking that it would be good to start on a simplified workflow for refactoring and test generation by leveraging LLMs to query over our codebase, pass to a code quality service, then chain and/or use prompt templates to perform relevant automations such as refactoring and writing tests that can be split into branches for manual review initially, then subsequently automated if all tests are passed.

Meanwhile, here’s how I am trying to organise the project on my Git repository which I am expecting to evolve further still so the below is still very much a work-in-progress but here’s what I’m thinking the different folders should handle:

• db - all database related operations such as CRUD

• demos - where some “stable” demos and workflows are defined

• gui - everything the user sees

• src - the bulk of the source code for our interfaces and implementations

• tests - unit and integration tests as well as fixtures for testing

I’m still missing the CI/CD, monitoring and evaluation components while the project structure is still being sorted out but I’m hoping to do some divide-and-conquer with other developers since I do need to deliver on “The Sound of Stories” (real-time, multilingual narration of generated stories) for the National Arts Council Arts x Tech Lab as well. If things go well though, these various developmental efforts should be highly synergistic and give me the building blocks for Berlayar A.I. the company. ;)

├── agents
│   └── __init__.py
├── dataloader.ipynb
├── db
│   ├── chunks.py
│   ├── connectors.py
│   ├── __init__.py
│   ├── main.py
│   ├── ops
│   │   ├── deeplake.py
│   │   └── __init__.py
│   └── utils.py
├── demos
│   ├── chat_adf
│   │   ├── chat.py
│   │   ├── tests.py
│   │   └── utils.py
│   ├── neural_search
│   │   └── upload.py
│   └── thesoundofstories
│       ├── audio
│       └── chat.py
├── diagram.py
├── gui
│   └── __init__.py
├── ingest_pdf.log
├── __init__.py
├── log_init.py
├── main.py
├── notebooks
│   ├── architecture.ipynb
│   ├── chat_adf.ipynb
│   └── dataloader.ipynb
├── poetry.lock
├── pyproject - Copy.toml
├── pyproject.toml

Meanwhile, putting this on the cold bench while I go off to focus on my upcoming exam for “Good Economics for Bad Times”.

Originally published on PubPub at erniesg.pubpub.org/pub/wpnv7rhs.