RafaelRuiz
/
assistance-engine
1
1
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'mrh-online-dev' of github.com:BRUNIX-AI/assistance-engine into mrh-online-dev

This commit is contained in:
pseco 2026-03-05 11:00:35 +01:00
parent 183c04829c d9d754bc6f
commit 010270bf22
38 changed files with 3697 additions and 68 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.vscode/settings.json vendored
View File

@ -7,10 +7,10 @@
"terminal.integrated.env.linux": { "terminal.integrated.env.linux": {
"PYTHONPATH": "${workspaceFolder}:${env:PYTHONPATH}" "PYTHONPATH": "${workspaceFolder}:${env:PYTHONPATH}"
}, },
"python.analysis.ignore": [ // "python.analysis.ignore": [
"*" // "*"
], // Disables Pylance's native linting // ], // Disables Pylance's native linting
"python.analysis.typeCheckingMode": "basic", // Keeps Pylance's type validation "python.analysis.typeCheckingMode": "off", // Keeps Pylance's type validation
"[python]": { "[python]": {
"editor.defaultFormatter": "charliermarsh.ruff", "editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true, "editor.formatOnSave": true,

59
README.md
View File

@ -43,14 +43,17 @@ graph TD
```text ```text
. .
├── Dockerfile # Container definition for the Engine ├── .github/
├── README.md # System documentation & Dev guide │ └── pull_request_template.md # Mandatory PR checklist
├── changelog # Version tracking and release history ├── Dockerfile # Container definition for the Engine
├── docker-compose.yaml # Local orchestration for dev environment ├── README.md # System documentation & Dev guide
├── CONTRIBUTING.md # Contribution standards & policies
├── changelog # Version tracking and release history
├── docker-compose.yaml # Local orchestration for dev environment
├── protos/ ├── protos/
│ └── brunix.proto # Protocol Buffers: The source of truth for the API │ └── brunix.proto # Protocol Buffers: The source of truth for the API
└── src/ └── src/
└── server.py # Core Logic: gRPC Server & RAG Orchestration └── server.py # Core Logic: gRPC Server & RAG Orchestration
``` ```
--- ---
@ -87,8 +90,6 @@ sequenceDiagram
Note over E: Close Langfuse Trace Note over E: Close Langfuse Trace
``` ```
--- ---
## Development Setup ## Development Setup
@ -102,15 +103,45 @@ sequenceDiagram
The engine utilizes Langfuse for end-to-end tracing and performance monitoring. The engine utilizes Langfuse for end-to-end tracing and performance monitoring.
1. Access the Dashboard: **http://45.77.119.180** 1. Access the Dashboard: **http://45.77.119.180**
2. Create a project and generate API Keys in **Settings**. 2. Create a project and generate API Keys in **Settings**.
3. Configure your local `.env` file: 3. Configure your local `.env` file using the reference table below.
### 3. Environment Variables Reference
> **Policy:** Every environment variable used by the engine must be documented in this table. Any PR that introduces a new variable without a corresponding entry here will be rejected. See [CONTRIBUTING.md](./CONTRIBUTING.md#5-environment-variables-policy) for full details.
Create a `.env` file in the project root with the following variables:
```env ```env
ELASTICSEARCH_URL=http://host.docker.internal:9200
ELASTICSEARCH_LOCAL_URL=http://localhost:9200
ELASTICSEARCH_INDEX=avap-docs-test
POSTGRES_URL=postgresql://postgres:postgres@localhost:5432/langfuse
LANGFUSE_HOST=http://45.77.119.180
LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-... LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=http://45.77.119.180 OLLAMA_URL=http://host.docker.internal:11434
OLLAMA_LOCAL_URL=http://localhost:11434
OLLAMA_MODEL_NAME=qwen2.5:1.5b
OLLAMA_EMB_MODEL_NAME=qwen3-0.6B-emb:latest
``` ```
### 3. Infrastructure Tunnels | Variable | Required | Description | Example |
|---|---|---|---|
| `ELASTICSEARCH_URL` | Yes | Elasticsearch endpoint used for vector/context retrieval in Docker | `http://host.docker.internal:9200` |
| `ELASTICSEARCH_LOCAL_URL` | Yes | Elasticsearch endpoint used for vector/context retrieval in local | `http://localhost:9200` |
| `ELASTICSEARCH_INDEX` | Yes | Elasticsearch index name used by the engine | `avap-docs-test` |
| `POSTGRES_URL` | Yes | PostgreSQL connection string used by the service | `postgresql://postgres:postgres@localhost:5432/langfuse` |
| `LANGFUSE_HOST` | Yes | Langfuse server endpoint (Devaron Cluster) | `http://45.77.119.180` |
| `LANGFUSE_PUBLIC_KEY` | Yes | Langfuse project public key for tracing and observability | `pk-lf-...` |
| `LANGFUSE_SECRET_KEY` | Yes | Langfuse project secret key | `sk-lf-...` |
| `OLLAMA_URL` | Yes | Ollama endpoint used for text generation/embeddings in Docker | `http://host.docker.internal:11434` |
| `OLLAMA_LOCAL_URL` | Yes | Ollama endpoint used for text generation/embeddings in local | `http://localhost:11434` |
| `OLLAMA_MODEL_NAME` | Yes | Ollama model name for generation | `qwen2.5:1.5b` |
| `OLLAMA_EMB_MODEL_NAME` | Yes | Ollama embeddings model name | `qwen3-0.6B-emb:latest` |
> Never commit real secret values. Use placeholder values when sharing configuration examples.
### 4. Infrastructure Tunnels
Open a terminal and establish the connection to the Devaron Cluster: Open a terminal and establish the connection to the Devaron Cluster:
```bash ```bash
@ -124,9 +155,7 @@ kubectl port-forward --address 0.0.0.0 svc/brunix-vector-db 9200:9200 -n brunix
kubectl port-forward --address 0.0.0.0 svc/brunix-postgres 5432:5432 -n brunix --kubeconfig ./kubernetes/ivar.yaml & kubectl port-forward --address 0.0.0.0 svc/brunix-postgres 5432:5432 -n brunix --kubeconfig ./kubernetes/ivar.yaml &
``` ```
### 5. Launch the Engine
### 4. Launch the Engine
```bash ```bash
docker-compose up -d --build docker-compose up -d --build
``` ```
@ -167,6 +196,8 @@ To maintain production-grade security and image efficiency, this project enforce
*Failure to comply with these architectural standards will result in PR rejection.* *Failure to comply with these architectural standards will result in PR rejection.*
For the full set of contribution standards, see [CONTRIBUTING.md](./CONTRIBUTING.md).
--- ---
## Security & Intellectual Property ## Security & Intellectual Property

134
ingestion/docs/10.1_Expressions.txt Normal file
View File

@ -0,0 +1,134 @@
6. Expressions in AVAP
This chapter explains the meaning of expression elements in AVAP.
6.1. Arithmetic Conversions
When describing an arithmetic operator in AVAP and using the phrase "numeric arguments are converted to a common type," it means that the operator's implementation for built-in types works as follows:
If either of the arguments is a complex number, the other is converted to complex.
Otherwise, if either of the arguments is a floating-point number, the other is converted to floating-point.
Otherwise, both must be integers, and no conversion is needed.
Additional rules may apply for certain operators.
6.2. Atoms
Atoms are the most basic elements of expressions in AVAP. The simplest atoms are identifiers or literals. Forms enclosed in parentheses, brackets, or braces are also syntactically categorized as atoms. The syntax for atoms is:
atom ::= identifier | literal | enclosure
enclosure ::= parenth_form | list_display | dict_display | set_display | generator_expression
6.2.1. Identifiers (Names)
An identifier that appears as an atom is a name. When the name is bound to an object, evaluating the atom yields that object. When a name is not bound, an attempt to evaluate it raises a NameError exception.
Private Name Mangling
When an identifier that occurs literally in a class definition begins with two or more underscores and does not end with two or more underscores, it is considered a private name of that class. Private names are transformed into a longer form before code is generated for them. The transformation inserts the class name, with the initial underscores removed and a single underscore inserted, in front of the name.
6.2.2. Literals
AVAP supports string and bytes literals, as well as various numeric literals:
literal ::= stringliteral | bytesliteral | integer | floatnumber | imagnumber
Evaluating a literal produces an object of the given type (string, bytes, integer, floating-point number, complex number) with the given value. All literals correspond to immutable data types.
6.2.3. Parenthesized Forms
A parenthesized form is an optional list of expressions enclosed in parentheses:
parenth_form ::= "(" [starred_expression] ")"
A parenthesized expression produces whatever the expression list produces: if the list contains at least one comma, it produces a tuple; otherwise, it produces the single expression that makes up the list of expressions.
6.2.4. Comprehensions for Lists, Sets and Dictionaries
To construct a list, set, or dictionary, AVAP provides special syntax called "comprehension," each in two flavors:
The contents of the container are listed explicitly.
They are computed using a set of loop and filtering instructions, called a "comprehension."
Common syntax elements for comprehensions are:
comprehension ::= assignment_expression comp_for
comp_for ::= "for" target_list "in" or_test [comp_iter]
comp_iter ::= comp_for | comp_if
comp_if ::= "if" or_test [comp_iter]
A comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the elements of the new container are those produced by considering each for or if clause as a block, nested from left to right, and evaluating the expression to produce an element each time the innermost block is reached.
6.2.5. List Displays
In AVAP, lists are generated and handled differently. To construct a list, the command variableToList(variable, list) is used, and an item from the list is retrieved with itemFromList(list, index, variable_to_store_item). To get the number of elements in the list, getListLen(list, var_to_store_list_length) is used.
The syntax for list displays is:
list_display ::= "[" [starred_list | comprehension] "]"
A list display produces a new list object, whose content is specified by a list of expressions or a comprehension. When a list of expressions is provided, its elements are evaluated from left to right and placed in the list object in that order.
6.2.6. Set Displays
A set display is denoted by curly braces and is distinguished from dictionary displays by the absence of colon characters separating keys and values:
set_display ::= "{" (starred_list | comprehension) "}"
A set display produces a new mutable set object, whose content is specified by a sequence of expressions or a comprehension.
6.2.7. Dictionary Displays
In AVAP, objects are created and managed using specific commands. An object is created with AddvariableToJSON(key, value, object_variable), and a key from the object is retrieved with variableFromJSON(object_variable, key, var_to_store_key_value).
The syntax for dictionary displays is:
dict_display ::= "{" [dict_item_list | dict_comprehension] "}"
dict_item_list ::= dict_item ("," dict_item)* [","]
dict_item ::= expression ":" expression | "**" or_expr
dict_comprehension ::= expression ":" expression comp_for
A dictionary display produces a new dictionary object. If a comma-separated sequence of dictionary items is provided, they are evaluated from left to right to define the dictionary entries.
Slices
A slice selects a range of elements in a sequence object (e.g., a string, tuple, or list). Slices can be used as expressions or as targets in assignments or statements. The syntax for a slice is as follows:
slicing ::= primary "[" slice_list "]"
slice_list ::= slice_item ("," slice_item)* [","]
slice_item ::= expression | proper_slice
proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]
lower_bound ::= expression
upper_bound ::= expression
stride ::= expression
There is ambiguity in the formal syntax here: anything that looks like a list expression also looks like a list slice, so any subscription might be interpreted as a slice. Instead of complicating the syntax further, this is disambiguated by defining that in this case, the interpretation as a subscription takes precedence over the interpretation as a slice (this is the case if the list slice does not contain a proper slice).
The semantics for a slice are as follows. The primary is indexed (using the same __getitem__() method as in a normal subscription) with a key constructed from the slice list, as follows. If the slice list contains at least one comma, the key is a tuple that contains the conversion of the slice elements; otherwise, the conversion of the single slice element is the key. The conversion of a slice element that is an expression is that expression. The conversion of a proper slice is a slice object whose start, stop, and step attributes are the values of the expressions given as the lower bound, upper bound, and step, respectively, substituting None for missing expressions.
Calls
A call invokes a callable object (e.g., a function) with a possibly empty series of arguments:
call ::= primary "(" [argument_list [","] | comprehension] ")"
argument_list ::= positional_arguments ["," starred_and_keywords]
["," keywords_arguments]
| starred_and_keywords ["," keywords_arguments]
| keywords_arguments
positional_arguments ::= positional_item ("," positional_item)*
positional_item ::= assignment_expression | "*" expression
starred_and_keywords ::= ("*" expression | keyword_item)
("," "*" expression | "," keyword_item)*
keywords_arguments ::= (keyword_item | "**" expression)
("," keyword_item | "," "**" expression)*
keyword_item ::= identifier "=" expression
An optional trailing comma may be present after positional and keyword arguments but does not affect the semantics.
The primary must evaluate to a callable object (user-defined functions, built-in functions, built-in object methods, class objects, class instance methods, and any object with a __call__() method are callable). All argument expressions are evaluated before attempting the call. Please refer to the Function Definitions section for the syntax of formal parameter lists.
If keyword arguments are present, they are first converted into positional arguments as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Then, for each keyword argument, the identifier is used to determine the corresponding slot. If the slot is already filled, a TypeError exception is raised. Otherwise, the argument is placed in the slot, filling it (even if the expression is None, it fills the slot). When all arguments have been processed, any slots that are still empty are filled with the default value from the function definition. If there are unfilled slots for which no default value is specified, a TypeError exception is raised. Otherwise, the list of filled slots is used as the argument list for the call.
Implementation Details in AVAP
In AVAP, variables are stored as strings, and lists and objects are managed using specific commands:
Lists: To generate a list, use variableToList(variable, list). To retrieve an item from the list, use itemFromList(list, index, variable_to_store_item). To get the number of items in the list, use getListLen(list, var_to_store_list_length).
Objects (dictionaries): An object is created with AddvariableToJSON(key, value, object_variable). To retrieve a key from the object, use variableFromJSON(object_variable, key, var_to_store_key_value).
Usage Example
Creation and management of lists:
// Creating a list
variableToList("item1", "myList")
variableToList("item2", "myList")
variableToList("item3", "myList")
// Retrieving an item from the list
itemFromList("myList", 1, "myVariable")
// Getting the length of the list
getListLen("myList", "listLength")
Creation and management of objects (dictionaries):
// Creating an object
AddvariableToJSON("key1", "value1", "myObject")
AddvariableToJSON("key2", "value2", "myObject")
// Retrieving a value by key from the object
variableFromJSON("myObject", "key1", "myVariable")
In this way, lists and objects in AVAP can be manipulated using the specific functions provided for working with variables stored as strings.

84
ingestion/docs/10.2_Operations.txt Normal file
View File

@ -0,0 +1,84 @@
Binary Arithmetic Operations
Binary arithmetic operations have the conventional levels of precedence. Some of these operations also apply to certain non-numeric types. Aside from the exponentiation operator, there are two levels: one for multiplicative operators and another for additive ones:
m_expr ::= u_expr | m_expr "*" u_expr | m_expr "@" m_expr |
m_expr "//" u_expr | m_expr "/" u_expr |
m_expr "%" u_expr
a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr
The * (multiplication) operator produces the product of its arguments. The arguments can both be numbers, or one argument must be an integer and the other a sequence. In the first case, the numbers are converted to a common type and then multiplied. In the second case, sequence repetition occurs; a negative repetition factor produces an empty sequence.
The @ (matrix multiplication) operator is intended for matrix multiplication. No built-in type in Python implements this operator.
The / (division) and // (floor division) operators produce the quotient of their arguments. Numeric arguments are converted to a common type. Division between integers produces a floating-point number, while floor division between integers results in an integer; the result is that of a mathematical division with the “floor” function applied to the result. Division by zero raises a ZeroDivisionError.
The % (modulus) operator produces the remainder of the division of the first argument by the second. Numeric arguments are converted to a common type. A zero argument on the right raises a ZeroDivisionError. Arguments can be floating-point numbers, e.g., 3.14 % 0.7 is equal to 0.34 (since 3.14 is equal to 4 * 0.7 + 0.34). The modulus operator always produces a result with the same sign as its second operand (or zero); the absolute value of the result is strictly smaller than the absolute value of the second operand.
The floor division and modulus operators are connected by the following identity: x == (x // y) * y + (x % y). Floor division and modulus are also connected by the built-in function divmod(): divmod(x, y) == (x // y, x % y).
In addition to performing the modulus operation on numbers, the % operator is also overloaded by string objects for old-style string formatting (also known as interpolation). The syntax for string formatting is described in the Python Library Reference, section Old-Style String Formatting.
The floor division operator, the modulus operator, and the divmod() function are not defined for complex numbers. Instead, convert to a floating-point number using the abs() function if appropriate.
The + (addition) operator produces the sum of its arguments. The arguments must both be numbers or both be sequences of the same type. In the first case, the numbers are converted to a common type and then added. In the second case, the sequences are concatenated.
The - (subtraction) operator produces the difference between its arguments. Numeric arguments are converted to a common type.
Shift Operations
Shift operations have lower precedence than arithmetic operations:
shift_expr ::= a_expr | shift_expr ("<<" | ">>") a_expr
These operators accept integers as arguments. They shift the first argument left or right by the number of bits specified by the second argument.
A right shift by n bits is defined as an integer floor division by pow(2, n). A left shift by n bits is defined as a multiplication by pow(2, n).
Binary Bitwise Operations
Each of the three binary bitwise operations has a different level of precedence:
and_expr ::= shift_expr | and_expr "&" shift_expr
xor_expr ::= and_expr | xor_expr "^" and_expr
or_expr ::= xor_expr | or_expr "|" xor_expr
* The & operator produces the bitwise AND of its arguments, which must be integers.
* The ^ operator produces the bitwise XOR (exclusive OR) of its arguments, which must be integers.
* The | operator produces the bitwise OR (inclusive OR) of its arguments, which must be integers.
Comparisons
Unlike C, all comparison operations in Python have the same priority, which is lower than any arithmetic, shift, or bitwise operation. Also, unlike C, expressions like a < b < c have the conventional mathematical interpretation:
comparison ::= or_expr (comp_operator or_expr)*
comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="
| "is" ["not"] | ["not"] "in"
Comparisons produce boolean values: True or False.
Comparisons can be arbitrarily chained, e.g., x < y <= z is equivalent to x < y and y <= z, except that y is evaluated only once.
Formally, if a, b, c, ..., y, z are expressions and op1, op2, ..., opN are comparison operators, then a op1 b op2 c ... y opN z is equivalent to a op1 b and b op2 c and ... y opN z, except that each expression is evaluated at most once.
Note that a op1 b op2 c does not imply any comparison between a and c, so, for example, x < y > z is perfectly legal.
Value Comparisons
The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects do not need to be of the same type.
The chapter Objects, Values, and Types states that objects have a value (in addition to type and identity). The value of an object is a rather abstract notion in Python: For example, there is no canonical method to access the value of an object. Furthermore, there is no requirement that the value of an object must be constructed in a particular way, e.g., composed of all its data attributes. Comparison operators implement a particular notion of what an object's value is.
The default behavior for equality comparison (== and !=) is based on object identity. Therefore, comparison of instances with the same identity results in equality, and comparison of equality of instances with different identities results in inequality.
No default comparison order (<, >, <=, >=) is provided; an attempt generates a TypeError.
The following list describes the comparison behavior of the most important built-in types:
Numbers: Built-in numeric types (int, float, complex) and types from the standard library (fractions.Fraction and decimal.Decimal) can be compared with themselves and among their types, with the restriction that complex numbers do not support order comparisons. Within the limits of the involved types, they are compared mathematically correctly without loss of precision.
None and NotImplemented: They are singletons. PEP 8 advises that comparisons for singletons should be done with is or is not, never with equality operators.
Binary Sequences: Instances of bytes or bytearray compare lexicographically using the numeric values of their elements.
Character Strings: Instances of str compare lexicographically using Unicode code points (the result of the built-in ord() function) or their characters.
Sequences: Instances of tuple, list, or range can only be compared within their types, with the restriction that ranges do not support order comparisons. Equality comparisons between these types result in inequality, and order comparisons between these types generate TypeError. They compare lexicographically using comparison of their corresponding elements.
Mappings: Instances of dict compare equal if and only if they have the same (key, value) pairs.
Sets: Instances of set or frozenset can be compared with each other and among their types. They define order comparison operators with the intention of checking subsets and supersets.
Other Built-in Types: Most other built-in types do not have comparison methods implemented, so they inherit the default comparison behavior.
User-defined classes that customize their comparison behavior should follow some consistency rules, if possible:
Equality comparison should be reflexive.
Comparison should be symmetric.
Comparison should be transitive.
If any of these conditions are not met, the resulting behavior is undefined.

157
ingestion/docs/10.3_Statements.txt Normal file
View File

@ -0,0 +1,157 @@
Simple Statements
In AVAP, a simple statement consists of a single logical line. Multiple simple statements can be placed on a single line, separated by semicolons. The syntax for simple statements is:
simple_stmt ::= expression_stmt | assert_stmt | assignment_stmt | augmented_assignment_stmt | annotated_assignment_stmt | pass_stmt | del_stmt | return_stmt | yield_stmt | raise_stmt | break_stmt | continue_stmt | import_stmt | future_stmt | global_stmt | nonlocal_stmt | type_stmt
Here’s a brief overview of each type of simple statement:
Expression Statement (expression_stmt): Executes an expression, which can be used for operations or calling functions.
Assert Statement (assert_stmt): Used for debugging purposes to test conditions.
Assignment Statement (assignment_stmt): Assigns values to variables or data structures.
Augmented Assignment Statement (augmented_assignment_stmt): Performs an operation on a variable and assigns the result back to the variable (e.g., x += 1).
Annotated Assignment Statement (annotated_assignment_stmt): Used for assigning values with annotations (e.g., type hints).
Pass Statement (pass_stmt): A placeholder that does nothing; used for syntactic requirements.
Del Statement (del_stmt): Deletes variables, items, or attributes.
Return Statement (return_stmt): Exits a function and optionally returns a value.
Yield Statement (yield_stmt): Produces a value from a generator function.
Raise Statement (raise_stmt): Raises exceptions for error handling.
Break Statement (break_stmt): Exits the closest enclosing loop.
Continue Statement (continue_stmt): Skips the current iteration of the closest enclosing loop.
Import Statement (import_stmt): Imports modules or specific components from modules.
Future Statement (future_stmt): Enables features from future versions of Python.
Global Statement (global_stmt): Declares variables as global within a function.
Nonlocal Statement (nonlocal_stmt): Declares variables as non-local, affecting scope in nested functions.
Type Statement (type_stmt): Declares or checks types (e.g., type hints).
Each simple statement performs a specific task and contributes to the overall functionality of the AVAP program.
Expression Statements
Expression statements are used (mostly interactively) to compute and write a value, or (usually) to call a method (a function that does not return a meaningful result; in Python, methods return the value None). Other uses of expression statements are allowed and occasionally useful. The syntax for an expression statement is:
expression_stmt ::= starred_expression
An expression statement evaluates the list of expressions (which can be a single expression).
In interactive mode, if the value is not None, it is converted to a string using the built-in function repr(), and the resulting string is written to the standard output on a line by itself (except if the result is None, in which case the called procedure produces no output).
Assignment Statements
Assignment statements in AVAP are used to (re)assign names to values and to modify attributes or elements of mutable objects. Here is the syntax:
assignment_stmt ::= (target_list "=")+ (starred_expression | yield_expression) target_list ::= target ("," target)* [","] target ::= identifier | "(" [target_list] ")" | "[" [target_list] "]" | attributeref | subscription | slicing | "*" target
Here's a breakdown of how assignment statements work:
Assignment Operation: An assignment statement evaluates the list of expressions and assigns the single resulting object to each of the target lists, from left to right.
Recursive Definition: The assignment operation is defined recursively depending on the form of the target list.
Target List: If the target list is a single object without ending in a comma, the object is assigned to that target. If the list contains a target prefixed with an asterisk, the object must be iterable with at least as many elements as targets, minus one. Elements before the starred target are assigned to the respective targets, and the remaining elements are assigned to the starred target.
Single Target: If the target is an identifier (name), it is bound to the object in the current local namespace. For other targets, names are bound in the global or enclosing namespace, depending on `nonlocal`.
Attribute Reference: If the target is an attribute reference, the primary expression is evaluated. It must produce an object with assignable attributes.
Subscription: If the target is a subscription, the primary expression is evaluated to produce a mutable sequence or mapping object, which is then used to assign the value.
Slice: If the target is a slice, the primary expression is evaluated, and the sequence object is requested to replace the slice with the assigned sequence elements.
In summary, assignment statements in AVAP are crucial for assigning values to variables and modifying data structures effectively.
Return Statement
The return statement in AVAP is used to return the value of a desired variable from a function. Here is the syntax:
return(variable_to_return):
Here is an overview of how the return statement works:
Function Context: The return statement can only occur within a function definition, not inside a nested class definition.
Variable Evaluation: If a variable is provided, it is evaluated. If no variable is specified, None is used by default.
Function Exit: The return statement exits the current function call and returns the specified value.
Interaction with try-finally: When the return statement is executed within a try statement that has a finally clause, the finally clause is executed before the function exits.
Generator Functions: In generator functions, the return statement indicates the end of the generator. It causes a StopIteration exception to be raised, with the returned value (if any) used to construct the StopIteration exception and set as the StopIteration.value attribute.
The return statement is a fundamental part of functions and generators, allowing for the output of values and proper function termination.
Raise Statement
In AVAP, the raise statement is used to throw an exception. The syntax for the raise statement is as follows:
raise [expression ["from" expression]]
If no expressions are present, raise re-raises the currently handled exception, also known as the active exception. If there is no active exception, a RuntimeError is raised indicating that it is an error.
Otherwise, raise evaluates the first expression as the exception object. It must be a subclass or an instance of BaseException. If it is a class, the exception instance is obtained when needed by creating an instance of the class without arguments.
The type of the exception is the instance of the exception class, and the value is the instance itself.
The from clause is used for exception chaining: if provided, the second expression must be another class or instance of exception. If the second expression is an exception instance, it will be attached to the raised exception as the __cause__ attribute (which is modifiable). If the expression is an exception class, the class will be instantiated and the resulting exception instance will be attached to the raised exception as the __cause__ attribute. If the raised exception is not handled, both exceptions will be printed.
startLoop() try: print(1 / 0) except Exception as exc: raise RuntimeError("Something went wrong") from exc endLoop()
A mechanism works implicitly if a new exception is raised while an exception is already being handled. An exception may be handled by an except or finally clause, or a with statement. The previous exception is then attached as the new exception’s __context__ attribute:
startLoop() try: print(1 / 0) except: raise RuntimeError("Something went wrong") from None endLoop()
Exception chaining can be explicitly suppressed by specifying None in the from clause:
startLoop() try: print(1 / 0) except: raise RuntimeError("Something went wrong") from None endLoop()
Break Statement
In AVAP, the break statement is used to terminate the closest enclosing loop. The syntax for the break statement is as follows:
break
When a break statement is encountered, it causes the loop to exit immediately, regardless of the loop's condition or any remaining iterations. This effectively transfers control to the statement following the loop.
The break statement is typically used within for or while loops to provide a way to exit the loop prematurely based on a certain condition.
for i in range(10): if i == 5: break print(i) print("Loop ended")
In this example, the loop will terminate when i equals 5, and "Loop ended" will be printed. The numbers 0 through 4 will be printed before the loop is exited.
Break Statement
The break statement in AVAP is used to terminate the closest enclosing loop. Here is an overview of its behavior:
Usage Context: The break statement can only occur within a for or while loop. It cannot be nested within a function or class definition inside that loop.
Loop Termination: It terminates the closest enclosing loop and skips the optional else clause if the loop has one.
Loop Control Target: If a for loop is terminated by break, the loop control target retains its current value.
Interaction with try-finally: When break is executed within a try statement with a finally clause, the finally clause is executed before actually exiting the loop.
The break statement is essential for controlling loop execution, allowing for early exit from loops and proper handling of loop cleanup.
Continue Statement
In AVAP, the continue statement is used to proceed with the next iteration of the closest enclosing loop. The syntax for the continue statement is as follows:
continue
The continue statement can only syntactically occur nested within a for or while loop, but not within a function or class definition inside that loop.
When continue is used within a loop that is also handling exceptions with a try statement containing a finally clause, the finally clause is executed before the next iteration of the loop begins.
for i in range(10): try: if i % 2 == 0: continue print(i) finally: print("In finally clause") print("Loop ended")
In this example, the continue statement will skip the current iteration when i is even, but before moving to the next iteration, the finally clause will print "In finally clause." For odd numbers, the loop will print the number and then "In finally clause." After the loop finishes, "Loop ended" will be printed.
Import Statement
In AVAP, the import statement is used to import an entire code file and define names in the local namespace. The syntax for the import statement is as follows:
import file.avap
The import statement in AVAP imports an entire code file and makes it available in the local namespace. No alias is assigned to the imported file; the file is simply referred to by its name.
For example:
# In the 'module.avap' file example_variable = 10 # In the main file import module.avap print(module.avap.example_variable) # Will print 10
In this example, the main file imports the module.avap file and can access the example_variable defined in that file using the module.avap syntax.
Compound Statements
In AVAP, compound statements contain (groups of) other statements; these affect or control the execution of those other statements in some way. In general, compound statements span multiple lines, though in simpler representations a complete compound statement might be contained within a single line.
if statements implement traditional flow control constructs. match specifies matching patterns for variable values. Function and class definitions are also syntactically compound statements.
A compound statement consists of one or more "clauses." A clause consists of a header and a "suite." The clause headers of a particular compound statement are all at the same level of indentation. Each clause header begins with a uniquely identifying keyword and ends with a colon. A suite is a group of statements controlled by a clause. A suite can be one or more simple statements separated by semicolons on the same line as the header, following the colon of the header, or it can be one or more statements indented on subsequent lines. Only the latter form of a suite can contain nested compound statements.
Control Flow Structures in AVAP
In AVAP, control flow structures include conditional statements and loops, which allow you to control the flow of execution based on conditions and iterate over a range of values.
If Statements
The syntax for an if statement in AVAP is:
if (variable, variableValue, comparator, expression): code to execute
This structure checks if the condition (variable compared to variableValue with the given comparator) is true, and if so, executes the block of code.
Loops
The syntax for a loop in AVAP is:
startLoop(variable, from, to) code to execute endLoop()
This structure initiates a loop where the variable iterates from the 'from' value to the 'to' value, executing the code block for each iteration.
The if Statement
The if statement in AVAP is used for conditional execution. The syntax is as follows:
if (variable, variableValue, comparator, expression): code to execute
This statement evaluates the condition specified by the variable, variableValue, comparator, and expression. It selects exactly one of the suites (blocks of code) by evaluating the expressions one by one until a true condition is found. The corresponding suite is then executed. If all conditions are false, no suites are executed.
The try Statement
The try statement in AVAP specifies exception handlers and/or cleanup code for a block of statements. The syntax is as follows:
try(): code to execute except(): code to execute
The try block contains code that might raise an exception. The except block contains code to handle exceptions raised by the try block. If an exception occurs, control is transferred to the except block. If no exception occurs, the except block is skipped.
Additional information about exceptions can be found in the section Exceptions, and information about using the raise statement to throw exceptions can be found in the section The raise Statement.

163
ingestion/docs/10.4_Patterns.txt Normal file
View File

@ -0,0 +1,163 @@
Patterns in AVAP
In AVAP, patterns provide a powerful way to match and destructure values. Patterns can be used in match statements to perform complex value comparisons and deconstructions. Here is a description of the available patterns and how they are used:
Literal Patterns: Match specific literal values such as numbers, strings, or booleans. For example:
match value: case 10: # Code to execute if value is 10 case "hello": # Code to execute if value is "hello"
Variable Patterns: Capture the value of a variable. This allows you to use the matched value in the corresponding case block:
match value: case x: # Code to execute, x will be assigned the value
Sequence Patterns: Match sequences like lists or tuples. You can also use the * operator to capture remaining elements:
match value: case [1, 2, *rest]: # Code to execute, rest will capture any additional elements
Mapping Patterns: Match dictionaries or similar mappings by specifying keys and their corresponding patterns:
match value: case "key": 42: # Code to execute if the dictionary has "key" with value 42
Class Patterns: Match instances of classes. You can also match specific attributes within the instance:
match value: case MyClass(attr1=42): # Code to execute if value is an instance of MyClass with attr1 equal to 42
Patterns in AVAP offer a flexible approach for handling different kinds of data structures and values, making it easier to write expressive and maintainable code.
OR Patterns
An OR pattern in AVAP allows you to specify multiple patterns separated by vertical bars (|). The OR pattern attempts to match each of its subpatterns with the subject value in order. If any of the subpatterns match, the OR pattern is considered successful. If none of the subpatterns match, the OR pattern fails.
or_pattern ::= "|".closed_pattern+
Here's how you can use OR patterns in practice:
match value: case 1 | 2 | 3: # Code to execute if value is 1, 2, or 3 case "hello" | "world": # Code to execute if value is "hello" or "world" case _: # Code to execute if value does not match any of the above
In this example:
The first case will match if value is either 1, 2, or 3.
The second case will match if value is either "hello" or "world".
The last case is a catch-all pattern that will execute if none of the previous patterns match.
OR patterns provide a concise way to handle multiple possible values or types, simplifying pattern matching and making your code more readable.
AS Patterns
An AS pattern in AVAP is used to bind an OR pattern to a name. This allows you to match a value with an OR pattern and simultaneously capture it under a specified name for further use. The syntax for an AS pattern is:
as_pattern ::= or_pattern "as" capture_pattern
When an AS pattern is used, if the OR pattern succeeds, the subject is bound to the name specified by the capture pattern, and the AS pattern itself succeeds.
Here's an example of how to use AS patterns:
match value: case 1 | 2 | 3 as x: print(f"Matched a number: x") case "hello" | "world" as greeting: print(f"Matched a greeting: greeting") case _: print("No match")
In this example:
The first case matches if value is 1, 2, or 3. The matched value is bound to the name x, which is then used in the print statement.
The second case matches if value is "hello" or "world". The matched value is bound to the name greeting, which is then used in the print statement.
The last case is a catch-all pattern that executes if none of the previous patterns match.
AS patterns are useful for capturing matched values under a name while using OR patterns, allowing for more flexible and readable pattern matching in your code.
Literal Patterns
In AVAP, literal patterns are used to match specific literal values, such as numbers, strings, or boolean values. The syntax for a literal pattern is:
literal_pattern ::= signed_number | strings | "None" | "True" | "False"
A literal pattern only succeeds if the value of the subject is equal to the specified literal value.
Here are examples of literal patterns and their usage:
match value: case 42: print("Matched the number 42") case "hello": print("Matched the string 'hello'") case None: print("Matched None") case True: print("Matched True") case False: print("Matched False") case _: print("No match")
In this example:
case 42: matches if value is exactly 42.
case "hello": matches if value is the string "hello".
case None: matches if value is None.
case True: matches if value is True.
case False: matches if value is False.
case _: is a catch-all pattern that executes if none of the previous patterns match.
Literal patterns are useful for matching specific, known values and are a fundamental part of pattern matching in AVAP.
Capture Patterns
In AVAP, capture patterns are used to bind the subject's value to a name. The syntax for a capture pattern is:
capture_pattern ::= NAME
Capture patterns always succeed and bind the value of the subject to the specified name.
Here’s how you might use capture patterns in AVAP:
match value: case x: print(f"Captured value: x")
In this example:
case x: captures whatever value is in value and binds it to the name x. The pattern always succeeds.
Capture patterns are useful when you want to extract and use the value of the subject within your code, regardless of what that value is.
Wildcard Patterns
In AVAP, wildcard patterns are used to match any value without binding it to a name. The syntax for a wildcard pattern is:
wildcard_pattern ::= '_'
Wildcard patterns always succeed and do not create any bindings. They are useful when you want to ignore the value of the subject and only care about whether it matches a certain pattern.
Here’s how you might use wildcard patterns in AVAP:
match value: case _: print("Matched any value")
In this example:
case _: matches any value and does not bind it to a name. The pattern always succeeds, and the code within this case will be executed regardless of the value.
Wildcard patterns are particularly useful when you need to handle a broad range of possibilities and are only interested in whether a value fits a general condition, not in the value itself.
Value Patterns
In AVAP, value patterns are used to match specific values. The syntax for a value pattern is:
value_pattern ::= attr
Value patterns only succeed if the subject's value matches the specified value. They are useful when you want to perform actions based on an exact value.
Here’s how you might use value patterns in AVAP:
match value: case 42: print("Matched the value 42") case "hello": print("Matched the string 'hello'") case _: print("Matched something else")
In this example:
case 42: matches the value 42 specifically.
case "hello": matches the string "hello" specifically.
case _: matches any other value not covered by the previous cases.
Value patterns are ideal for scenarios where you need to check for specific values and respond accordingly. They provide precise control over the matching process.
Group Patterns
In AVAP, group patterns are used to group multiple patterns together. The syntax for a group pattern is:
group_pattern ::= "(" pattern ")"
Group patterns are useful when you want to combine patterns or when patterns need to be evaluated together. They have the same effect as the pattern they contain but allow for more complex pattern structures.
Here’s an example of how to use group patterns in AVAP:
match value: case (42 | 43): print("Matched either 42 or 43") case (name, age) if age > 18: print(f" is an adult") case _: print("Matched something else")
In this example:
case (42 | 43): uses a group pattern to match either the value 42 or 43.
case (name, age) if age > 18: uses a group pattern to match a tuple and includes an additional condition on the age.
case _: matches any other value not covered by the previous cases.
Group patterns are ideal for creating more complex matching scenarios where patterns need to be combined or grouped together.
Sequence Patterns
In AVAP, sequence patterns are used to match elements within sequences like lists or tuples. The syntax for sequence patterns is:
sequence_pattern ::= "[" [maybe_sequence_pattern] "]" | "(" [open_sequence_pattern] ")"
Sequence patterns can match elements of sequences based on specific rules. Here’s how they work:
List Patterns: Use square brackets [ ] to match lists. You can include patterns for the elements within the list.
case [a, b, c]: print("Matched a list with three elements")
Tuple Patterns: Use parentheses ( ) to match tuples. Similarly, you can specify patterns for the tuple elements.
case (x, y): print("Matched a tuple with two elements")
Sequence patterns allow for flexible and powerful matching of sequence types. They can match sequences of various lengths and structures by defining the pattern for each element.
Here’s an example of using sequence patterns in a match statement:
match value: case [1, 2, 3]: print("Matched a list with elements 1, 2, 3") case (a, b, c) if a + b == c: print("Matched a tuple where a + b equals c") case _: print("Matched something else")
In this example:
case [1, 2, 3]: matches a list with exactly the elements 1, 2, and 3.
case (a, b, c) if a + b == c: matches a tuple and includes a condition to check if a + b equals c.
case _: matches any other value not covered by the previous cases.
Mapping Patterns
In AVAP, mapping patterns are used to match mapping elements, such as dictionaries. Here is the syntax and behavior of mapping patterns:
mapping_pattern ::= { [items_pattern] }
Mapping Patterns are designed to match elements within mappings, such as dictionaries. They use specific rules to determine if a pattern matches the given mapping.
Syntax: Mapping patterns are enclosed in curly braces { ... }. The items_pattern specifies the pattern for the mapping items.
Matching Rules: The rules for matching mapping patterns include checking for key-value pairs in the mapping and ensuring they align with the specified pattern.
Usage: Mapping patterns are useful for destructuring dictionaries and other mapping types in a concise manner.
Mapping patterns enhance pattern matching capabilities by allowing for specific and flexible matching of dictionary elements.
Class Patterns
In AVAP, class patterns are used to match instances of specific classes. Here is a detailed overview:
class_pattern ::= name "(" [pattern_arguments ","?] ")"
Pattern Syntax: A class pattern specifies the class name followed by a parenthesized list of pattern_arguments. The pattern matches instances of the specified class.
Matching Instances: The pattern will match if the subject is an instance of the specified class and the pattern_arguments (if any) match according to the rules defined for the pattern.
Usage: Class patterns are useful for deconstructing objects based on their class and extracting values from them, enabling more precise pattern matching.
These patterns provide a way to work with objects based on their class type and structure, facilitating more sophisticated pattern matching and value extraction.

108
ingestion/docs/10_Execution_model_in_avap.txt Normal file
View File

@ -0,0 +1,108 @@
Execution Model in AVAP
4.1. Structure of a Program
A program in AVAP is built from code blocks that execute linearly. A block is a section of the AVAP program text that executes as a unit. Code blocks in AVAP include:
A script file.
The body of a function.
An import statement for additional files.
Each line of code in AVAP is considered a block and executes sequentially. There is no interactive execution, deferred execution, or object classes.
4.2. Names and Bindings
4.2.1. Name Binding
Names in AVAP refer to values and are introduced through name binding operations. The following constructs bind names:
Formal parameters of functions.
Function definitions.
Assignment expressions.
Name binding is performed using the addVar(value, variable) function, which assigns the value to the specified variable. There are no class declarations or complex targets in AVAP. Only functions and direct assignments to variables are valid code blocks.
4.2.2. Name Resolution
A scope defines the visibility of a name in a code block. In AVAP, if a variable is defined in a code block, its scope includes that block. The scope of a variable within a function extends to the entire function block.
When a name is used in a code block, it is resolved using the nearest enclosing scope. If the name is not found in the current scope, a NameError exception is raised.
If a name binding operation occurs anywhere within a code block, all uses of the name within that block are treated as references to the current block. This means that variables must be defined before their use within the same block.
In AVAP, there are no global or nonlocal declarations. All names are resolved within the scope in which they are defined. There is no dynamic code execution with eval or exec, so all bindings must be static and known at code writing time.
4.3. Importing Files
In AVAP, it is possible to import the contents of other code files. The import file.avap statement inserts the contents of the specified file at the exact point where the import statement appears. This process is linear and sequential, meaning that the imported content is executed as if it were part of the original file.
It is crucial that the necessary functions are defined before they are called. If a function is not defined before its call, a NameError exception will be raised.
Example of import usage:
avap
// Content of the file main.avap
addVar(10, x)
import functions.avap
myFunction(x)
// Content of the file functions.avap
def myFunction(y):
addVar(y + 5, result)
print(result)
4.4. Exceptions
Exceptions in AVAP allow for the handling of errors or exceptional conditions. An exception is raised when an error is detected; it can be handled by the surrounding code block or by any code block that directly or indirectly invoked the block where the error occurred.
The AVAP interpreter raises an exception when it detects a runtime error. An AVAP program can also explicitly raise an exception using the raise statement. Exception handlers are specified with the try ... except statement.
Example of exception handling:
try:
addVar(10 / 0, result)
except ZeroDivisionError:
print("Cannot divide by zero.")
In this example, if a division by zero occurs, a ZeroDivisionError exception is raised and handled by the except block.
This structure ensures that AVAP programs execute in a sequential and predictable manner, without advanced dynamic or deferred execution features, maintaining simplicity and clarity in name binding and import handling.
5. The Import System in AVAP
AVAP code in one file gains access to code in another file through the import process. The import statement is the only way to invoke the import machinery in AVAP.
The import statement inserts the contents of the specified file at the exact point where the import statement appears in the original file. There are no other ways to invoke the import system in AVAP.
When an import statement is executed, the contents of the imported file are processed as if they were part of the original file, ensuring that all functions and variables from the imported file are available in the context of the original file. If the specified file is not found, a FileNotFoundError is raised.
Example of using the import statement in AVAP:
Content of file main.avap
addVar(10, x)
import functions.avap
myFunction(x)
Content of file functions.avap
def myFunction(y):
addVar(y + 5, result)
print(result)
In this example, the content of functions.avap is inserted into main.avap at the point of the import statement, ensuring that myFunction is defined before being called.
5.1. Import Rules
Position of Import: The import statement must be placed at the exact location where the content of the imported file is to be included. The content of the imported file is executed linearly along with the original file.
Import Error: If the file specified in the import statement is not found, a FileNotFoundError is raised.
Scope of Imports: The functions and variables from the imported file are added to the local scope of the original file at the point of import. This means they can be accessed as if they were defined in the same file.
5.2. Limitations and Considerations
No Packages: Unlike other languages, AVAP does not have a hierarchical package system. Each file is imported independently and treated as an autonomous unit.
Sequential Execution: Execution in AVAP is sequential and does not allow lazy or deferred execution. Therefore, all functions and variables must be defined before use, and the content of imported files must be in the correct order.
No Conditional Import: The import statement in AVAP does not support conditions. The specified file will always be imported at the point of the statement, regardless of any conditions.
5.3. Advanced Example
Consider the following example where multiple files are imported:
Content of the file main.avap
addVar(5, a)
import utilities.avap
import operations.avap
addVar(utilities.increment(a), b)
addVar(operations.multiply(b, 2), c)
print(c)
Content of the file utilities.avap
def increment(x):
return x + 1
Content of the file operations.avap
def multiply(x, y):
return x * y
In this example, utilities.avap and operations.avap are imported into main.avap at the specified points, allowing the increment and multiply functions to be used in main.avap.

39
ingestion/docs/11_Conditional_statements.txt Normal file
View File

@ -0,0 +1,39 @@
IF-THEN-ELSE Statement
The IF-THEN-ELSE statement in AVAP™ allows for decision-making based on specific conditions and executes different blocks of code depending on the outcome of those conditions. Below is a detailed explanation of its syntax and functionality.
6.1 Syntax of the IF-THEN-ELSE Statement
The basic syntax of the IF-THEN-ELSE statement in AVAP™ is as follows:
IF(condition, true_value, operator)
// Block of code if the condition is true ELSE
// Block of code if the condition is false END()
condition: This is an expression that evaluates to either true or false.
true_value: This is the value assigned if the condition is true.
operator: This is the operator used to compare the condition with the true value.
6.2 Functioning of the IF-THEN-ELSE Statement
The IF-THEN-ELSE statement evaluates the given condition and, if it is true, executes the block of code within the IF(). If the condition is false, it executes the block of code within the ELSE().
Below is the description of each part of the IF-THEN-ELSE statement using the provided example:
// IF, ELSE and END Sample Use
addVar(selector,'yes')
IF(selector,'yes','=')
addVar(result,1) ELSE()
addVar(result,0) END()
addResult(result)
The variable selector is initialized with the value 'yes'.
The statement IF(selector,'yes','=') evaluates whether the value of selector is equal to 'yes'. In this case, the condition is true.
Inside the IF() block, addVar(result,1) is executed, which assigns the value 1 to the result variable.
Since the condition of the IF() is true, the code block inside the ELSE() is not executed.
The statement addResult(result) adds the value of the result variable to the API result.
6.3 Result
The result returned by the API after executing the above code is as follows:
{
status
, elapsed:0.008270740509033203, result: { result:1 }
}
This result indicates that the execution was successful (status:true) and that the value of result is 1.
6.4 Conclusions
The IF-THEN-ELSE statement in AVAP™ provides an efficient way to make decisions based on specific conditions. Similar to other programming languages, it allows for executing different blocks of code based on the outcome of evaluating a condition.

42
ingestion/docs/12_Loop_statement.txt Normal file
View File

@ -0,0 +1,42 @@
StartLoop() Statement
The loop statement in AVAP™ allows you to execute a block of code repeatedly until a specific condition is met. Below is a detailed explanation of its syntax and functionality.
7.1 Syntax of the Loop Statement
The full syntax of the loop statement in AVAP™ is as follows:
startLoop(control, start, end) // Code block to repeat endLoop()
This syntax consists of three main parts:
control: This is the loop control variable used to track the progress of the loop. It is initialized with the starting value of the loop and is incremented with each iteration until it reaches the end value.
start: This is the starting value of the loop. The loop begins at this value.
end: This is the ending value of the loop. The loop terminates when the control variable reaches this value.
7.2 Functioning of the Loop Statement
The loop statement in AVAP™ follows this execution process:
The control variable control is initialized with the starting value specified in start.
The loop condition is evaluated: while the value of control is less than or equal to the end value end, the code block within startLoop() is executed. If the value of control exceeds the end value, the loop terminates, and execution continues after endLoop().
In each iteration of the loop, the code block within startLoop() is executed, and the control variable control is automatically incremented by one.
Once the control variable reaches or exceeds the end value end, the loop terminates, and execution continues after endLoop().
7.3 Example of Use
Below is an example of using the loop statement in AVAP™, along with a detailed explanation of each part of the code:
// Loop Sample Use
// Initialize the variable 'variable' with the value 5.
addVar(variable,5)
// Start the loop with the control variable 'control', ranging from 1 to 5.
startLoop(control,1,5)
// In each iteration of the loop, assign the current value of 'control' to the variable 'counter'.
addVar(counter,$control)
endLoop()
// Add the final value of 'counter' to the API result.
addResult(counter)
7.4 Result and Conclusions
After executing the above code, the result returned by the API is as follows:
{
status
, elapsed:0.01605510711669922, result: { counter:5 }
}
This result confirms that the execution was successful (status:true) and that the final value of counter is 5.
In summary, the loop statement in AVAP™ provides an efficient way to execute a block of code repeatedly within a specified range. By automating tasks that require repetition, such as processing a list of items or generating sequential numbers, this statement becomes a fundamental tool for programming in AVAP™.

26
ingestion/docs/13_Api_inbound_interface.txt Normal file
View File

@ -0,0 +1,26 @@
Chapter 12: addParam() Function
Introduction
The addParam() function in AVAP™ is a powerful tool used to add parameters to an API call in the query string. This parameter is assigned to a variable and acts as a bridge between the API call and the API itself, allowing smooth and efficient communication between both.
Usage of addParam
The addParam() function is used to add parameters to an API call in the query string. The basic syntax of this function is as follows:
addParam(variable, value)
Where variable is the name of the variable to be used as a parameter in the API call, and value is the value assigned to this variable.
Example Usage
Below is a practical example illustrating how to use the addParam() function in an API call:
# API call with addParam()
addParam(user, "john_doe")
addParam(password, "password123")
In this example, two parameters, user and password, are being added to an API call. The value of user is set to "john_doe" and the value of password is set to "password123".
Internal Operation
Internally, the addParam() function constructs the querystring for the API call by adding the specified parameters along with their corresponding values. This querystring is passed to the API, which uses it to process the request and return the appropriate response.
Important Considerations
It is important to ensure that the parameters added with addParam() are valid and correctly formatted according to the requirements of the API being called. Additionally, it is the developer's responsibility to ensure that the values assigned to the parameters are secure and do not contain malicious data that could compromise system security.
Conclusions
The addParam() function in AVAP™ is an essential tool for constructing and managing API calls, facilitating communication between the client and the server. By understanding how this function works and how it is used in the context of an API call, developers can create more robust and secure applications that make the most of web services' potential.

42
ingestion/docs/14_Working_with_libraries.txt Normal file
View File

@ -0,0 +1,42 @@
Function Libraries
Introduction
Includes are a fundamental feature in AVAP™ that allow for the efficient organization and reuse of code in software development projects. Just like in other programming languages, includes in AVAP™ enable the incorporation of functionalities from other files or libraries into the current file. This capability provides a number of significant advantages that make the development and maintenance of projects more efficient and effective.
Purpose of Includes
The primary purpose of includes in AVAP™ is to promote modularity and code reuse. By dividing code into separate modules or files and then including them in main files as needed, developers can write and maintain code in a more organized and structured manner. This facilitates the management of large and complex projects, as well as collaboration between development teams.
Advantages of Using Includes
Code Reuse: Includes allow for the reuse of functions, variables, and other code definitions in multiple parts of a project, reducing code duplication and promoting consistency and coherence in development.
Facilitates Maintainability: By dividing code into smaller, more specific modules, it is easier to identify, understand, and modify parts of the code without affecting other parts of the project. This eases software maintenance over time.
Promotes Modularity: The ability to include files selectively as needed encourages code modularity, which simplifies understanding and managing complex projects by breaking them down into smaller, manageable components.
Improves Readability and Organization: The use of includes helps organize code in a logical and structured manner, improving readability and facilitating navigation through different parts of the project.
Syntax of Includes
In AVAP™, the syntax for including a file is similar to that of other languages like C. The keyword include is used followed by the name of the file to be included. There are two main ways to include files in AVAP™:
Local Include: Used to include project-specific files located in the same directory or in subdirectories relative to the current file. The file name is specified within quotes. Example:
include "file_name.avap"
System Include: Used to include standard or system library files located in predefined or configured paths on the system. The file or library name is specified between angle brackets (< and >). Example:
include <library_name.avap>
Operation
When an include is found in an AVAP™ file, the interpreter searches for the specified file and incorporates it into the current file at compile time. This means that all the code contained in the included file will be available for use in the current file.
Common Uses
Including Standard Libraries: Standard libraries that provide common functions and utilities can be included to simplify application development.
Including Definition Files: Files containing definitions of variables, constants, or data structures used in multiple parts of the project can be included.
Including Specific Functionality Modules: Modules providing additional features for the project, such as file handling, text processing, or data manipulation, can be included.
Practical Example
Suppose we have a file named utils.avap that contains utility functions we want to use in our main project. We can include this file in our main project as follows:
include "utils.avap" // We can now use the functions defined in utils.avap
With this understanding of the value and advantages of using includes in AVAP™, we will explore in detail their operation and practical application in project development.
Practical Example
Suppose we have a file named utils.avap that contains utility functions we want to use in our main project. We can include this file in our main project as follows:
include "utils.avap" // We can now use the functions defined in utils.avap
With this understanding of the value and advantages of using includes in AVAP™, we will explore in detail their operation and practical application in project development.
Function Libraries Function Products
In AVAP™, there are a series of function libraries grouped by categories called Function Products that complement the base AVAP™ language and leverage the power of AVS servers for distribution. Through Function Products, developers can extend the functionality of AVAP™ by incorporating specialized libraries tailored to different needs and applications.
Function Products provide a way to access advanced features and capabilities not available in the core language, offering a robust framework for building complex and scalable solutions. These libraries are designed to integrate seamlessly with AVAP™, enhancing the development process and enabling more efficient and effective project execution.

34
ingestion/docs/15_Function_declaration.txt Normal file
View File

@ -0,0 +1,34 @@
Function Declaration
Introduction
Functions in AVAP™ are reusable blocks of code that perform a specific task. Just like in Python, functions in AVAP™ allow for code modularization, improved readability, easier maintenance, and code reuse.
Function Construction
In AVAP™, similar to Python, functions are defined using the keyword def, followed by the function name and its parameters in parentheses. The function definition ends with a colon (:), followed by the block of code that forms the function body, indented with four spaces.
Defining a function in AVAP™
def greet(name):
return "Hello, " + name + "!"
Calling the function
message = greet("World")
print(message)
Output
Hello, World!
Technical Features
Parameters: Functions can accept zero or more parameters that are used as inputs to the function.
Return Values: Functions can return a value using the return keyword.
Scope: Functions in AVAP™ have their own scope, meaning that variables defined within a function are only visible within that function unless declared as global variables.
Code Reusability: Functions allow for encapsulating and reusing blocks of code that perform specific tasks.
Practical Example
Below is a practical example illustrating the definition and invocation of a function in AVAP™:
Definition of a Function to Calculate the Area of a Circle
def calculate_circle_area(radius):
return 3.14 * radius ** 2
Calling the Function
circle_radius = 5
area = calculate_circle_area(circle_radius)
print("The area of the circle is:", area)
Output:
The area of the circle is: 78.5
Conclusions
Functions are a fundamental part of programming in AVAP™, allowing for effective organization and modularization of code. By understanding how to define, construct, and call functions in AVAP™, developers can write clearer, more concise, and maintainable code, facilitating the development and management of applications.

721
ingestion/docs/16_Appendix.txt Normal file
View File

@ -0,0 +1,721 @@
Appendix
Function Glossary
randomString()
The randomString() command generates a random string based on a specified pattern and stores it in a target variable. It is especially useful when random strings are needed to conform to a specific format, such as passwords or identifiers.
Parameters
Pattern
Type: var
Description: A regular expression (regex) pattern that defines the characters and structure of the string to be generated. It can be a direct value or a variable containing the pattern. For example, [a-zA-Z0-9] will generate a string that includes uppercase letters, lowercase letters, and numbers.
Length
Type: var
Description: An integer value specifying the length of the random string to be generated. It can be a direct value or a variable containing the desired length. This value determines how many characters the resulting string will have.
TargetVariable
Type: var
Description: The variable where the generated string will be stored. This variable should be used later in the program. Unlike the other parameters, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
randomString('[a-zA-Z0-9]', 8, generatedPassword)
// Call using variables:
pattern = '[a-zA-Z0-9]'
length = 8
randomString(pattern, length, generatedPassword)
stampToDatetime()
The stampToDatetime() command converts a timestamp value to a date and time according to a specified format, applying a possible time difference, and stores the result in a target variable. It is useful for manipulating and formatting time values into different representations.
Parameters
timestamp
Type: var
Description: A value representing a timestamp, which can be provided directly or through a variable. This value is the starting point for conversion to a date and time format.
Format
Type: var
Description: A format string that defines how the resulting date and time should be presented. This string follows the same conventions used in Python for formatting dates and times. Common symbols include:
%Y: Year with four digits (e.g., 2024)
%m: Month with two digits (01 to 12)
%d: Day of the month with two digits (01 to 31)
%H: Hour in 24-hour format (00 to 23)
%M: Minutes (00 to 59)
%S: Seconds (00 to 59)
For example, the format %Y-%m-%d %H:%M:%S converts a timestamp into a string like 2024-08-25 14:30:00. It can be a direct value or a variable containing the desired format.
TimeDelta
Type: var
Description: An optional value representing a time adjustment (positive or negative) applied to the timestamp before conversion. This value can be provided directly or through a variable and is expressed in seconds.
TargetVariable
Type: var
Description: The variable where the resulting date and time from the conversion will be stored. Unlike the other parameters, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
stampToDatetime(1692966600, '%Y-%m-%d %H:%M:%S', 3600, convertedDatetime)
// Call using variables:
timestamp = 1692966600
format = '%Y-%m-%d %H:%M:%S'
adjustment = 3600
stampToDatetime(timestamp, format, adjustment, convertedDatetime)
In the first example, a timestamp is converted to a date and time in the format "%Y-%m-%d %H:%M:%S", applying a 3600-second (1-hour) adjustment, and the result is stored in the variable convertedDatetime. In the second example, variables are used to define the timestamp, format, and adjustment.
getTimeStamp()
The getTimeStamp() command converts a date and time string, given in a specific format, to a timestamp value. Additionally, it allows for an optional time adjustment before storing the result in a target variable. This command is useful for converting human-readable date and time representations to a numeric timestamp format, which can be used in calculations or time comparisons.
Parameters
DateString
Type: var
Description: A string representing a date and time. This string must follow the format specified in the Format parameter. It can be a direct value or a variable containing the date string.
Format
Type: var
Description: A format string that defines how to interpret the date and time string (DateString). This string follows Python's conventions for formatting and parsing dates and times. Some common symbols include:
%Y: Year with four digits (e.g., 2024)
%m: Month with two digits (01 to 12)
%d: Day of the month with two digits (01 to 31)
%H: Hour in 24-hour format (00 to 23)
%M: Minutes (00 to 59)
%S: Seconds (00 to 59)
For example, to interpret the string "2024-08-25 14:30:00", the format %Y-%m-%d %H:%M:%S would be used. It can be a direct value or a variable containing the format.
TimeDelta
Type: var
Description: An optional value representing a time adjustment (positive or negative) applied to the timestamp after conversion. This value can be provided directly or through a variable and is expressed in seconds.
TargetVariable
Type: var
Description: The variable where the resulting timestamp from the conversion will be stored. Unlike the other parameters, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
getTimeStamp('2024-08-25 14:30:00', '%Y-%m-%d %H:%M:%S', 3600, generatedTimestamp)
// Call using variables:
date = '2024-08-25 14:30:00'
format = '%Y-%m-%d %H:%M:%S'
adjustment = 3600
getTimeStamp(date, format, adjustment, generatedTimestamp)
In the first example, the date and time string "2024-08-25 14:30:00" is converted to a timestamp, applying a 3600-second (1-hour) adjustment, and the result is stored in the variable generatedTimestamp. In the second example, variables are used to define the date, format, and adjustment.
getRegex()
The getRegex() command searches for matches in a source string using a regular expression (regex) pattern and stores the result in a target variable. This command is useful for extracting specific parts of a string that match a defined pattern, such as email addresses, phone numbers, or any other structure defined by a regex.
Parameters
SourceVariable
Type: variable
Description: The variable containing the source string in which to search for regex pattern matches. This string is the text on which the regex search will be applied.
rePattern
Type: variable
Description: The variable containing the regular expression (regex) pattern that defines what to search for in the source string. This pattern should follow standard regex rules, allowing the specification of sequences of characters to identify in the source string.
TargetVariable
Type: variable
Description: The variable where the search result will be stored. Depending on the context and the pattern used, the result could be the first match found, all matches, or even specific groups within the match.
Usage Example
// Direct call with values:
sourceText = "Email: user@example.com and phone: 123-456-7890"
pattern = r"\b\d{3}-\d{3}-\d{4}\b"
getRegex(sourceText, pattern, phoneNumber)
// Call using variables:
sourceText = "Visit our website at https://www.example.com for more information."
regexPattern = r"https?://\S+"
getRegex(sourceText, regexPattern, foundURL)
In the first example, a phone number in the format 123-456-7890 is searched in the sourceText string and the result is stored in the phoneNumber variable. In the second example, a URL is extracted from the sourceText string using a regex that identifies URL patterns, and the result is stored in the foundURL variable.
getDateTime()
The getDateTime() command retrieves the current date and time, formats it according to a specified format, applies an optional time adjustment, and converts it to a specific time zone before storing the result in a target variable. It is useful for obtaining and manipulating the current date and time in different formats and time zones.
Parameters
Format
Type: var
Description: A format string that defines how the resulting date and time should be presented. This string follows the date and time formatting conventions used in Python. Some of the most common symbols include:
%Y: Year with four digits (e.g., 2024)
%m: Month with two digits (01 to 12)
%d: Day of the month with two digits (01 to 31)
%H: Hour in 24-hour format (00 to 23)
%M: Minutes (00 to 59)
%S: Seconds (00 to 59)
For example, the format "%Y-%m-%d %H:%M:%S" will present the date and time as 2024-08-25 14:30:00. It can be a direct value or a variable containing the desired format.
TimeDelta
Type: var
Description: An optional value representing a time adjustment (positive or negative) applied to the current date and time before conversion. This value can be provided directly or through a variable and is expressed in seconds.
TimeZone
Type: var
Description: The time zone to which the date and time should be converted. This value can be a time zone identifier provided directly or through a variable. Some common time zones include:
"UTC": Coordinated Universal Time
"America/New_York": U.S. Eastern Time (EST/EDT)
"America/Los_Angeles": U.S. Pacific Time (PST/PDT)
"Europe/London": London Time (GMT/BST)
"Europe/Madrid": Madrid Time (CET/CEST)
"Asia/Tokyo": Tokyo Time (JST)
"Australia/Sydney": Sydney Time (AEST/AEDT)
You can use any time zone recognized by the pytz library in Python, which includes most time zones worldwide.
TargetVariable
Type: var
Description: The variable in which the resulting date and time from the operation will be stored. Unlike the other parameters, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
getDateTime('%Y-%m-%d %H:%M:%S', 3600, 'UTC', currentTime)
// Call using variables:
format = '%Y-%m-%d %H:%M:%S'
adjustment = 3600
timeZone = 'America/New_York'
getDateTime(format, adjustment, timeZone, currentDateTime)
In the first example, the current date and time are retrieved, adjusted by 3600 seconds (1 hour), converted to UTC, and stored in the variable currentTime. In the second example, variables are used to define the format, time adjustment, and time zone, with the result stored in the currentDateTime variable.
encodeMD5()
The encodeMD5() command generates an MD5 hash of the provided string and stores the result in a target variable. MD5 is a cryptographic hash function that produces a 128-bit value (32 hexadecimal characters), commonly used to verify data integrity.
Parameters
SourceVariable
Type: var
Description: The variable containing the text string to be encoded in MD5. It can be a direct value or a variable storing the input string.
TargetVariable
Type: var
Description: The variable in which the resulting MD5 hash will be stored. Unlike the SourceVariable parameter, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
encodeMD5('example_string', md5Hash)
// Call using variables:
text = 'example_string'
hashVariable = 'md5Hash'
encodeMD5(text, hashVariable)
In the first example, an MD5 hash is generated from the string 'example_string' and stored in the md5Hash variable. In the second example, a variable text is used to define the input string and another variable hashVariable is used to store the resulting MD5 hash.
encodeSHA256()
The encodeSHA256() command generates a SHA-256 hash of the provided string and stores the result in a target variable. SHA-256 is a cryptographic hash function that produces a 256-bit value (64 hexadecimal characters), offering greater security compared to MD5.
Parameters
SourceVariable
Type: var
Description: The variable containing the text string to be encoded in SHA-256. It can be a direct value or a variable storing the input string.
TargetVariable
Type: var
Description: The variable in which the resulting SHA-256 hash will be stored. Unlike the SourceVariable parameter, this must be a variable and not a direct value.
Usage Example
// Direct call with values:
encodeSHA256('example_string', sha256Hash)
// Call using variables:
text = 'example_string'
hashVariable = 'sha256Hash'
encodeSHA256(text, hashVariable)
In the first example, a SHA-256 hash is generated from the string 'example_string' and stored in the sha256Hash variable. In the second example, a variable text is used to define the input string, and another variable hashVariable is used to store the resulting SHA-256 hash.
getQueryParamList()
The getQueryParamList() command extracts the query parameters from the current HTTP request and stores a list of these parameters in a target variable. This is useful for handling and processing query parameters in web applications.
Parameters
TargetVariable
Type: var
Description: The variable in which the extracted query parameter list will be stored. This should be a variable where the command's result will be saved.
Command Flow
Parameter Extraction: Accesses the query parameters from the current HTTP request.
List Construction: Creates a list containing dictionaries, where each dictionary represents a query parameter and its associated value.
Result Storage: Saves the list of parameters in the variable specified by TargetVariable.
Usage Example
Suppose the HTTP query has the following parameters: ?user=alice&age=30.
// Define the variable to store the result
queryParamsList = []
// Call the command to extract query parameters
getQueryParamList(queryParamsList)
// Return the list of query parameters via addResult
addResult(queryParamsList)
Given the query string ?user=alice&age=30, the getQueryParamList() command will generate the following list of parameters:
[
{"user": "alice"},
{"age": "30"}
]
getListLen()
The getListLen() command calculates the length of a list and stores the result in a target variable. This command is useful for determining the number of elements in a list.
Parameters
SourceVariable
Type: var
Description: The variable containing the list whose length you want to calculate. It can be a variable that stores the list or a direct value representing the list.
TargetVariable
Type: var
Description: The variable where the result of the list length will be stored. This should be a variable that will receive the integer value representing the number of elements in the list.
Command Flow
Retrieve the List: Access the list stored in the SourceVariable.
Calculate the Length: Calculate the number of elements in the list.
Store the Result: Save the calculated length in the variable specified by TargetVariable.
Usage Example
Suppose the list in myList is ['apple', 'banana', 'cherry'].
// Variable definitions
myList = ['apple', 'banana', 'cherry']
listLength = 0
// Call the command to calculate the length of the list
getListLen(myList, listLength)
// Return the list length through addResult
addResult(listLength)
Since the list myList has 3 elements, the getListLen() command will calculate that the length is 3. This value will be stored in the listLength variable and returned through addResult(listLength), resulting in the following output:
3
itemFromList()
The itemFromList() command extracts a specific element from a list based on a given index and stores the result in a target variable. This is useful for accessing individual elements within a list.
Parameters
SourceVariable
Type: var
Description: The variable containing the list from which an element is to be extracted. It can be a variable that stores the list or a direct value representing the list.
index
Type: value
Description: The index of the element to be extracted from the list. It must be an integer value that indicates the position of the element within the list.
TargetVariable
Type: var
Description: The variable where the extracted element will be stored. It must be a variable that will receive the value of the element at the specified index position.
Command Flow
Access the List: Access the list stored in the SourceVariable.
Extract the Element: Retrieve the element at the position specified by the index.
Store the Result: Save the extracted element in the variable specified by TargetVariable.
Usage Example
Suppose the list in myList is ['apple', 'banana', 'cherry'] and you want to extract the element at index 1.
// Variable definitions
myList = ['apple', 'banana', 'cherry']
element = ''
// Call the command to extract the element at index 1
itemFromList(myList, 1, element)
// Return the extracted element through addResult
addResult(element)
Since index 1 corresponds to the element 'banana' in the myList, the itemFromList() command will extract 'banana' and store it in the variable element. The element variable will be returned through addResult(element), resulting in the following output:
"banana"
variableFromJSON()
The variableFromJSON() command extracts the value associated with a specific key from a JSON object and stores the result in a target variable. This command is useful for accessing values within a JSON object.
Parameters
SourceVariable
Type: var
Description: The variable containing the JSON object from which a value is to be extracted. It can be a variable that stores the JSON object or a direct value representing the JSON object.
key
Type: value
Description: The key whose value is to be extracted from the JSON object. It must be a value that represents the key within the JSON object.
TargetVariable
Type: var
Description: The variable where the extracted value will be stored. It must be a variable that will receive the value associated with the specified key in the JSON object.
Command Flow
Access the JSON Object: Access the JSON object stored in the SourceVariable.
Extract the Value: Retrieve the value associated with the key within the JSON object.
Store the Result: Save the extracted value in the variable specified by TargetVariable.
Usage Example
Suppose the JSON object in jsonData is "name": "Alice", "age": 30 and you want to extract the value associated with the key "name".
// Variable definitions
jsonData = {"name": "Alice", "age": 30}
nameValue = ''
// Call the command to extract the value associated with the key "name"
variableFromJSON(jsonData, "name", nameValue)
// Return the extracted value through addResult
addResult(nameValue)
Since the value associated with the key "name" in the JSON object jsonData is "Alice", the variableFromJSON() command will extract "Alice" and store it in the variable nameValue. The nameValue variable will be returned through addResult(nameValue), resulting in the following output:
"Alice"
AddVariableToJSON()
The AddVariableToJSON() command adds a new key and its corresponding value to a JSON object and stores the result in a target variable. This command is useful for updating a JSON object with new key-value pairs.
Parameters
Key
Type: variable
Description: The key to be added to the JSON object. It must be a variable that stores the key to be added.
Value
Type: variable
Description: The value associated with the key to be added to the JSON object. It must be a variable that stores the corresponding value.
TargetVariable
Type: variable
Description: The variable where the updated JSON object will be stored. It must be a variable that will receive the JSON object with the new key and its added value.
Command Flow
Access the JSON Object: Access the JSON object stored in the TargetVariable.
Add the Key and Value: Add the new key and its associated value to the JSON object.
Store the Result: Save the updated JSON object in the variable specified by TargetVariable.
Usage Example
Suppose the initial JSON object in jsonData is "name": "Alice", "age": 30, and you want to add a new key "email" with the value "alice@example.com".
// Variable definitions
jsonData = {"name": "Alice", "age": 30}
newKey = "email"
newValue = "alice@example.com"
// Call the command to add the new key and value to the JSON object
AddVariableToJSON(newKey, newValue, jsonData)
// Return the updated JSON object through addResult
addResult(jsonData)
This updated JSON object will be stored in the variable jsonData and will be returned through addResult(jsonData), resulting in the following output:
{
"name": "Alice",
"age": 30,
"email": "alice@example.com"
}
variableToList()
The variableToList() command converts an element into a list that contains only that element and stores the resulting list in a target variable. This command is useful to ensure that a single value is handled as a list in subsequent processing.
Parameters
element
Type: variable
Description: The variable that contains the element to be converted into a list. It can be any type of value that you want to include as the only item in the list.
TargetVariable
Type: variable
Description: The variable in which the resulting list will be stored. It must be a variable that will receive the list with the included element.
Command Flow
Access the Element: Access the element stored in the element variable.
Create the List: Create a list that contains only the provided element.
Store the Result: Save the resulting list in the variable specified by TargetVariable.
Usage Example
Suppose the element in myElement is "apple" and you want to convert it into a list.
// Variable definitions
myElement = "apple"
myList = []
// Call the command to convert the element into a list
variableToList(myElement, myList)
// Return the resulting list through addResult
addResult(myList)
Since myElement is "apple", the variableToList() command will convert this element into a list with a single item: ["apple"]. This list will be stored in the variable myList, and myList will be returned through addResult(myList), resulting in the following output:
["apple"]
addParam()
The addParam() command retrieves the value associated with a specific key from the query string of the current request and assigns this value to a target variable. This command is useful for extracting values from query parameters in an HTTP request and storing them in variables for processing.
Parameters
param
Type: value
Description: The key of the query string whose value you want to retrieve. It should be a value that represents the key in the query string.
variable
Type: var
Description: The variable in which the retrieved value from the query string will be stored. It must be a variable that will receive the value associated with the specified key.
Command Flow
Retrieve the Value: Access the value associated with the param key from the query string of the current request.
Assign the Value: Assign the retrieved value to the variable specified by variable.
Usage Example
Suppose the query string of the current request is ?user=alice&age=30, and you want to retrieve the value associated with the key "user".
// Variable definitions
userName = ''
// Call the command to retrieve the value for the "user" key and assign it to the variable
addParam("user", userName)
// Return the retrieved value through addResult
addResult(userName)
Given the query string ?user=alice&age=30, the addParam() command will retrieve the value "alice" associated with the key "user" and store it in the userName variable. The userName variable will be returned through addResult(userName), resulting in the following output:
"alice"
addResult()
The addResult() command is used to return the content of a variable as part of the command or function response. It is the way to present results or processed data from commands and operations performed in the language.
Parameters
variable
Type: var
Description: The variable whose content is to be returned as the result. It should be a variable that contains the value or data you want to include in the response.
Command Flow
Access the Content: Access the content of the variable provided as a parameter.
Return the Result: Include the content of the variable in the final response.
Example Usage
Suppose we have performed an operation and want to return the result stored in the result variable.
// Define the variable with the result of an operation
result = "Operation completed successfully."
// Call the command to return the content of the variable
addResult(result)
In this example, the addResult(result) command will return the content of the result variable, which is "Operation completed successfully.". This content will be presented as part of the response.
Note
The addResult() command is the primary mechanism for returning information and results in the language. Make sure that the variable passed to the command contains the desired data or result before calling addResult().
RequestPost()
The RequestPost() command performs an HTTP POST request to a specified URL, sending a query string, headers, and a request body, and stores the result of the request in a destination variable. This command is useful for sending data to a server and handling the responses from the request.
Parameters
url
Type: variable
Description: The URL to which the POST request will be sent. It should be a variable containing the address of the resource to which the request is to be made.
querystring
Type: variable
Description: The query string that will be appended to the URL. It should be a variable containing the query parameters in string format.
headers
Type: variable
Description: The HTTP headers that will be included in the POST request. It should be a variable containing a dictionary of headers and their values.
body
Type: variable
Description: The body of the POST request that will be sent to the server. It should be a variable containing the data to be sent in the request.
o_result
Type: variable
Description: The variable in which the result of the POST request will be stored. It should be a variable that will receive the server's response.
Command Flow
Build the Request: Uses the provided URL, query string, headers, and body to construct the POST request.
Send the Request: Sends the POST request to the specified server.
Store the Result: Saves the server's response in the variable specified by o_result.
Example Usage
Suppose you want to send a POST request to https://api.example.com/data, with a query string userId=123, headers including Content-Type: application/json, and a body with JSON data.
// Define variables
url = "https://api.example.com/data"
querystring = "userId=123"
headers = {"Content-Type": "application/json"}
body = '{"name": "Alice", "age": 30}'
response = ''
// Call the command to perform the POST request
RequestPost(url, querystring, headers, body, response)
// Return the request result via addResult
addResult(response)
In this example, the RequestPost() command will send a POST request to https://api.example.com/data with the provided query string, headers, and body. The server's response will be stored in the response variable, and this variable will be returned via addResult(response). The result of the request will be included in the final response.
ormCreateTable()
The ormCreateTable() command creates a new table in a database using the specified ORM (Object-Relational Mapping). This command defines the columns of the table and their data types, and stores a reference to the created table in a destination variable.
Parameters
fields
Type: value
Description: A string containing the names of the table columns, separated by commas. Each column name should correspond to a field in the table.
fieldsType
Type: value
Description: A string containing the data types for each column, separated by commas. The data types should be in the same order as the column names in fields.
dbaseName
Type: value
Description: The name of the database where the table will be created. It should be a string indicating the target database.
varTarget
Type: variable
Description: The variable in which the reference to the created table will be stored. It should be a variable that will receive the reference to the new table.
Command Flow
Define the Table: Uses the column names (fields) and their data types (fieldsType) to define the structure of the new table.
Create the Table: Creates the table in the database specified by dbaseName using the provided definition.
Store the Result: Saves the reference to the created table in the variable specified by varTarget.
Example Usage
Suppose you want to create a table called users in a database called myDatabase, with two columns: username of type VARCHAR and age of type INTEGER.
// Define variables
fields = "username,age"
fieldsType = "VARCHAR,INTEGER"
dbaseName = "myDatabase"
tableReference = ''
// Call the command to create the table
ormCreateTable(fields, fieldsType, dbaseName, tableReference)
// Return the reference to the created table via addResult
addResult(tableReference)
In this example, the ormCreateTable() command will create a table in the myDatabase database with the specified columns and data types. The reference to the new table will be stored in the tableReference variable, and this variable will be returned via addResult(tableReference). The output will include the reference to the created table.
ormCheckTable()
The ormCheckTable() command checks for the existence of a table in a specific database and stores the result in a destination variable. This command is useful for verifying if a table already exists before attempting further operations on it.
Parameters
dbaseName
Type: value
Description: The name of the database in which the table's existence should be checked. It should be a string indicating the database to check.
varTarget
Type: variable
Description: The variable in which the result of the check will be stored. It should be a variable that will receive a value indicating whether the table exists or not.
Command Flow
Check Existence: Accesses the database specified by dbaseName to verify if the requested table exists.
Store the Result: Saves the result of the check in the variable specified by varTarget. The stored value will indicate whether the table exists (True or False).
Example Usage
Suppose you want to check if a table called users exists in a database called myDatabase.
// Define variables
dbaseName = "myDatabase"
tableExists = ''
// Call the command to check the existence of the table
ormCheckTable(dbaseName, tableExists)
// Return the result of the check via addResult
addResult(tableExists)
In this example, the ormCheckTable() command will check for the existence of the users table in the myDatabase database. The result of the check (whether the table exists or not) will be stored in the tableExists variable, and this variable will be returned via addResult(tableExists). The output will reflect whether the table exists (True) or not (False).
ormAccessUpdate()
The ormAccessUpdate() command updates records in a database table based on the provided selection criteria. This command modifies the values of specified fields in a database using the corresponding values from variables.
Parameters
fields
Type: variable
Description: A string containing the names of the fields to be updated. The field names should be separated by commas.
fieldsValuesVariables
Type: variable
Description: A string containing the names of the variables holding the new values for the specified fields. The variable names should be separated by commas, in the same order as the fields in fields.
dbase
Type: variable
Description: The name of the database where the table to be updated is located. It should be a variable containing the name of the database.
selector
Type: variable
Description: A condition to select the records to be updated. It should be a string specifying the selection criteria in SQL format, such as id = 1.
varTarget
Type: variable
Description: The variable in which the result of the update operation will be stored. It should be a variable that will receive a value indicating whether the update was successful or not.
Command Flow
Define Fields and Values: Uses the field names (fields) and the variables with the values to be updated (fieldsValuesVariables) to define which records should be modified and with what data.
Select Records: Uses the condition provided in selector to identify the records to be updated.
Update the Database: Performs the update in the database specified by dbase, applying the changes to the records that meet the selector condition.
Store the Result: Saves the result of the update operation in the variable specified by varTarget. The stored value will indicate whether the update was successful (True) or failed (False).
Example Usage
Suppose you want to update the age field to 31 for the user with id equal to 1 in a database called myDatabase.
// Define variables
fields = "age"
fieldsValuesVariables = "newAge"
dbase = "myDatabase"
selector = "id = 1"
updateSuccess = ''
// Define the variable holding the new value
newAge = 31
// Call the command to update the record
ormAccessUpdate(fields, fieldsValuesVariables, dbase, selector, updateSuccess)
// Return the result of the update via addResult
addResult(updateSuccess)
In this example, the ormAccessUpdate() command will update the age field in the myDatabase database for the record where id = 1. The new value for age is 31, stored in the newAge variable. The updateSuccess variable will store the result of the operation (whether it was successful or not), and this variable will be returned via addResult(updateSuccess).
ormAccessSelect()
The ormAccessSelect() command retrieves records from a table in a database based on the provided selection criteria. This command selects the desired fields and stores the results in a target variable.
Parameters
fields
Type: variable
Description: A string containing the names of the fields to be retrieved. The field names should be separated by commas.
dbase
Type: variable
Description: The name of the database from which records should be retrieved. It must be a variable containing the name of the database.
selector
Type: variable
Description: A condition to select the records to be retrieved. It must be a string specifying the selection criteria in SQL format, such as id = 1.
varTarget
Type: variable
Description: The variable in which the query results will be stored. It must be a variable that will receive a list of dictionaries, each representing a retrieved record.
Command Flow
Defining the Fields: Use the field names (fields) to specify which data should be retrieved.
Selecting Records: Use the condition provided in selector to identify which records should be selected from the database.
Retrieving Data: Access the database specified by dbase and retrieve the records that meet the selector condition, including only the specified fields.
Storing the Result: Save the query results in the variable specified by varTarget. The stored value will be a list of dictionaries, where each dictionary represents a retrieved record with the requested fields.
Example Usage
Suppose you want to retrieve the username field for all users where age is greater than 25 from a database called myDatabase.
// Define variables
fields = "username"
dbase = "myDatabase"
selector = "age > 25"
usersList = ''
// Call the command to retrieve the records
ormAccessSelect(fields, dbase, selector, usersList)
// Return the query results via addResult
addResult(usersList)
In this example, the ormAccessSelect() command will retrieve the username field for all users in the myDatabase database where age is greater than 25. The results will be stored in the usersList variable, and this variable will be returned via addResult(usersList). The output will be a list of dictionaries, each representing a user whose username has been retrieved.
ormAccessInsert()
The ormAccessInsert() command inserts a new record into a database table using the provided values for the fields. This command defines the fields and their corresponding values, and stores the result of the operation in a target variable.
Parameters
fields
Type: variable
Description: A string containing the names of the fields into which the values will be inserted. The field names should be separated by commas.
fieldsValuesVariables
Type: variable
Description: A string containing the names of the variables that hold the values to be inserted into the specified fields. The variable names should be separated by commas, in the same order as the fields in fields.
dbase
Type: variable
Description: The name of the database where the table into which the new record should be inserted is located. It must be a variable containing the name of the database.
varTarget
Type: variable
Description: The variable in which the result of the insertion operation will be stored. It must be a variable that will receive a value indicating whether the insertion was successful or not.
Command Flow
Defining the Fields and Values: Use the field names (fields) and the variables with the values to be inserted (fieldsValuesVariables) to define what data should be inserted.
Inserting into the Database: Perform the insertion of the new record into the database specified by dbase, using the provided values.
Storing the Result: Save the result of the insertion operation in the variable specified by varTarget. The stored value will indicate whether the insertion was successful (True) or failed (False).
Example Usage
Suppose you want to insert a new record into a table called users in a database called myDatabase, with values for username and age coming from the variables newUsername and newAge.
// Define variables
fields = "username,age"
fieldsValuesVariables = "newUsername,newAge"
dbase = "myDatabase"
insertSuccess = ''
// Define the variables with the new values
newUsername = "Alice"
newAge = 31
// Call the command to insert the new record
ormAccessInsert(fields, fieldsValuesVariables, dbase, insertSuccess)
// Return the result of the insertion via addResult
addResult(insertSuccess)
In this example, the ormAccessInsert() command will insert a new record into the myDatabase database in the users table. The values for username and age are provided by the newUsername and newAge variables. The insertSuccess variable will store the result of the operation (whether it was successful or not), and this variable will be returned via addResult(insertSuccess). The output will reflect whether the insertion was successful (True) or failed (False).
ormAI()
The ormAI() command uses an artificial intelligence model to convert a natural language query into an SQL statement, which is then executed against a database. This command processes a natural language query to generate an SQL statement that is executed on the table specified in the source parameter, and stores the result in a target variable.
Parameters
prompt
Type: variable
Description: A string in natural language that describes the query to be made. For example, "get the value of the row with id 5".
source
Type: variable
Description: The name of the table on which the generated query should be executed. It must be a variable containing the name of the table in the database.
TargetVariable
Type: variable
Description: The variable in which the result of the query will be stored. It must be a variable that will receive the result of the generated and executed SQL query.
Command Flow
Generating SQL Query: Use the artificial intelligence model to convert the prompt into an SQL statement. For example, if the prompt is "get the value of the row with id 5", the AI will generate the SQL query SELECT * FROM source WHERE id = 5;.
Executing the Query: Execute the generated SQL statement on the table specified in source.
Storing the Result: Save the result of the query execution in the variable specified by TargetVariable. The result will be the dataset retrieved by the executed SQL statement.
Example Usage
Suppose you want to retrieve all the data from the row with id equal to 5 from a table called users.
// Define variables
prompt = "get the value of the row with id 5"
source = "users"
queryResult = ''
// Call the command to process the query
ormAI(prompt, source, queryResult)
// Return the query result via addResult
addResult(queryResult)
In this example, the ormAI() command will convert the prompt into an SQL query: SELECT * FROM users WHERE id = 5;. This query will be executed on the users table, and the results will be stored in the queryResult variable. The queryResult variable will be returned via addResult(queryResult). The output will be the dataset retrieved by the executed SQL statement.
functionAI()
The functionAI() command uses an artificial intelligence model to convert a natural language description of a function or process into a code implementation, which is then executed and returns the result. This command converts a description provided in prompt into a function that operates on the data of the table specified in source, and stores the result in a target variable.
Parameters
prompt
Type: variable
Description: A string in natural language that describes the process or function to be executed. For example, "calculate the average of the salary column".
source
Type: variable
Description: The name of the table on which the generated function should be executed. It must be a variable containing the name of the table in the database.
TargetVariable
Type: variable
Description: The variable in which the result of the executed function or process will be stored. It must be a variable that will receive the result of the generated and executed code.
Command Flow
Generating Code: Use the artificial intelligence model to convert the prompt into a code implementation. For example, if the prompt is "calculate the average of the salary column", the AI will generate the code necessary to calculate the average of that column.
Executing the Code: Execute the generated code on the table specified in source.
Storing the Result: Save the result of the code execution in the variable specified by TargetVariable. The result will be the calculated value or the dataset produced by the executed code.
Example Usage
Suppose you want to calculate the average of the salary column in a table called employees.
// Define variables
prompt = "calculate the average of the salary column"
source = "employees"
averageSalary = ''
// Call the command to process the function
functionAI(prompt, source, averageSalary)
// Return the result of the function via addResult
addResult(averageSalary)
In this example, the functionAI() command will convert the prompt into a code implementation to calculate the average of the salary column in the employees table. The result of the calculation will be stored in the averageSalary variable, and this variable will be returned via addResult(averageSalary). The output will be the calculated average of the salary column.

140
ingestion/docs/17_Architecture_memory_foundations.txt Normal file
View File

@ -0,0 +1,140 @@
SECTION I: Architecture, Memory, and Foundations
This section establishes the foundations of how AVAP manages service logic and in-memory data manipulation. Unlike conventional interpreted languages, AVAP uses a hybrid evaluation engine that enables the combination of declarative commands with dynamic expressions.
1.1 Endpoint Registration (registerEndpoint)
The registerEndpoint command is the atomic configuration unit. It acts as the bridge between the network layer (HTTP) and the application code.
Interface
registerEndpoint(path, method, middleware, description, handler, output)
Parameter Specification
path (String):
Defines the URL route. Supports static routes and is designed for future implementations of route parameters (variable segments).
method (String):
Specifies the allowed HTTP verb (GET, POST, PUT, DELETE). The server will automatically reject any request that does not match this method (Error 405).
middleware (List):
A list of functions executed sequentially before the handler. Ideal for JWT token validation or maintenance checks. If any middleware function fails, execution stops before reaching the main business logic.
description (String):
Metadata for automatic documentation generation (Swagger/OpenAPI). It does not affect execution but is critical in the development lifecycle.
handler (Function):
The logical entry point. This is the name of the main function where the business logic resides.
output (Variable):
Defines the “master” variable that the engine will automatically return at the end of execution, unless additional results are specified via addResult.
1.2 The Variable Assignment Engine (Dynamic Assignment)
AVAP allows direct assignment syntax using the = symbol, providing flexibility similar to languages such as Python, but under strict contextual control.
Internal Mechanics: The eval Process
When the interpreter encounters an instruction of the form variable = expression, it triggers a three-step process:
Cleanup and Tokenization:
The engine determines whether the expression contains references to existing variables (using $), method calls, or literals.
Expression Evaluation:
Operations are resolved in real time. This enables:
Boolean Logic:
is_valid = (age > 18 and has_permission == True)
Arithmetic:
tax = subtotal * 0.21
String Formatting:
query = "SELECT * FROM users WHERE id = %s" % retrieved_id
Object and Property Resolution:
Allows deep access to complex structures returned by database connectors or APIs:
customer_email = user_list[0].profile.email
Memory Impact
Unlike addVar, dynamic assignment can transform the variable’s type at runtime (Mutable Type System). If a variable originally contained a number and is later assigned a string after evaluation, the engine automatically updates the variable’s metadata.
1.3 State Initialization and References (addVar)
addVar is the fundamental command for defining the global script state.
Interface
addVar(targetVarName, varValue)
Advanced Behavior
Intelligent Automatic Typing:
The engine inspects varValue. If it detects a numeric format (even if provided as a string from configuration), it internally converts it to int or float. It supports both commas and periods interchangeably, normalizing the value for mathematical operations.
The $ Reference Prefix:
This is the dereferencing operator.
addVar(copy, $original)
Instructs the engine not to assign the literal string "$original", but instead to look up the current value of the variable original in the symbol table and copy it.
Scope:
Variables created with addVar in the main body of the script are considered Request Session Variables. This means they persist throughout the lifecycle of that specific API call execution, but remain isolated from other concurrent requests to ensure data safety (thread-safety).
Syntax Summary
Syntax Usage Description
name = "John" Direct Assignment Creates a simple string.
total = $price * 1.10 Dynamic Evaluation Uses the value of price in a calculation and stores the result.
addVar(status, 200) Initialization Explicit method to ensure creation in the global context.
data = res[0].info Object Access Extracts a specific property from a JSON object or DB result.
Examples
1. Hello World
(State Initialization – Section 1.3)
addVar(message, "Hello world from AVAP")
addResult(message)
2. Mathematical Assignment
(Dynamic Assignment & Arithmetic Evaluation – Section 1.2)
subtotal = 150.50
tax = subtotal * 0.21
total = subtotal + tax
addResult(total)
3. Dynamic String Concatenation
(Expression Evaluation – String Formatting – Section 1.2)
name = "System"
log = "Event registered by: %s" % name
addResult(log)
4. Value Reference ($)
(State Initialization & Dereferencing – Section 1.3)
addVar(base, 1000)
addVar(copy, $base) // copy takes the value 1000, not the string "$base"
addResult(copy)
5. Boolean Assignment
(Boolean Expression Evaluation – Section 1.2)
level = 5
is_admin = level >= 10
addResult(is_admin) // Returns False
6. Multiple Response (JSON Construction)
(Global State Initialization – Section 1.3)
addVar(code, 200)
addVar(status, "Success")
addResult(code)
addResult(status)
// Result: {"code": 200, "status": "Success"}

151
ingestion/docs/18_Input_output_management.txt Normal file
View File

@ -0,0 +1,151 @@
SECTION II: Input and Output (I/O) Management
This section describes the mechanisms AVAP uses for external data ingestion, parameter integrity validation, and construction of the response payload delivered to the final client.
2.1 Intelligent Parameter Capture (addParam)
The addParam command is responsible for extracting information from the incoming HTTP request. Its design is source-agnostic, simplifying development by not requiring the programmer to specify where the data originates.
Interface
addParam(param_name, target_variable)
Priority Mechanism (Cascading Search)
When addParam is invoked, the AVAP engine inspects the request in the following hierarchical order:
Query Arguments:
Parameters present in the URL (e.g., ?id=123).
JSON Body:
If the request includes Content-Type: application/json, the engine searches for the key inside the JSON object.
Form Data / Body Arguments:
Data submitted via standard forms (x-www-form-urlencoded).
Technical Behavior
Automatic Decoding:
The engine attempts to decode values into ASCII/UTF-8 format, eliminating encoding inconsistencies.
Null Handling:
If the requested parameter does not exist in any source, the target variable is initialized as None. This enables subsequent security checks using if blocks.
2.2 Collection Validation and Counting (getListLen)
To ensure API robustness, it is necessary to validate the volume of received information. getListLen acts as AVAP’s volume inspector.
Interface
getListLen(source_variable, target_variable)
I/O Applications
Parameter Validation:
Counts how many elements are contained in a variable populated by addParam or getQueryParamList.
Loop Safety:
Before initiating a startLoop, it is recommended to use getListLen to define the upper bound of the iteration, preventing overflow errors.
Database Results:
After a query, determines whether records were retrieved (length > 0) or if the result set is empty.
2.3 Multiple List Capture (getQueryParamList)
There are scenarios where the same parameter is sent multiple times (e.g., search filters such as ?color=red&color=blue). AVAP manages this through specialized list-based capture.
Interface
getQueryParamList(param_name, target_list_variable)
Effect
Transforms all occurrences of param_name into a structured list within target_list_variable. If only one value is present, a single-element list is created. This ensures downstream logic can always treat the data as a collection, preventing type errors.
2.4 Response Construction (addResult)
The addResult command registers which variables will be included in the response body. AVAP dynamically constructs an output JSON object based on calls to this command.
Interface
addResult(source_variable)
Advanced Features
Promise Handling:
If the variable passed to addResult is the result of an operation initiated with go_async, the engine automatically marks that field as "promised" in the response or returns the thread ID if synchronization has not completed.
String Cleanup:
The engine detects redundant quotation marks (resulting from prior evaluations) and normalizes them to ensure the resulting JSON is valid and clean.
Multi-Registration:
Multiple calls to addResult are allowed. Each call adds a new key to the output JSON object. By default, the JSON key matches the variable name, unless a custom key is defined in the engine.
2.5 HTTP Status Control (_status)
AVAP uses a reserved system variable to communicate with the underlying web server and define the success or error code of the transaction.
Using _status
By assigning a numeric value to the _status variable (using addVar or direct assignment), the programmer defines the HTTP response code.
Code Common Usage in AVAP
200 Successful operation (default value).
201 Resource successfully created.
400 Parameter validation error (Bad Request).
401 Authentication failure.
500 Internal error caught inside an exception block.
Integrated Example of Section II
// Capture input
addParam("user_id", id)
// Validate presence
if(id, None, '==')
addVar(_status, 400)
addVar(error, "User ID is required")
addResult(error)
return()
end()
// If execution reaches here, respond with success
addVar(_status, 200)
addResult(id)
Examples
1. ID Capture
(Input Ingestion – Section 2.1)
addParam("client_id", internal_id)
// If the client sends ?client_id=500, internal_id will be 500
addResult(internal_id)
2. Parameter Counter
(Collection Validation – Section 2.2)
addParam("data_list", my_list)
getListLen(my_list, count)
addResult(count)
3. Null Validation
(Input Validation & HTTP Status Control – Sections 2.1 & 2.5)
addParam("api_key", key)
if(key, None, "==")
addVar(_status, 403)
addVar(error, "Access denied: missing API KEY")
addResult(error)
end()
4. Multiple List Capture
(Multi-Value Query Capture – Section 2.3)
getQueryParamList("emails", email_list)
addResult(email_list)
5. Multiple Response (JSON Construction)
(Response Construction – Section 2.4)
addVar(code, 200)
addVar(status, "Success")
addResult(code)
addResult(status)
// Result: {"code": 200, "status": "Success"}

192
ingestion/docs/19_Control_logic_decision_structures.txt Normal file
View File

@ -0,0 +1,192 @@
SECTION III: Control Logic and Decision Structures
This section details how AVAP manages execution flow. The language uses explicitly closed block structures that enable clear sequential reading, simplifying the debugging of complex APIs.
3.1 The Conditional Block (if / else / end)
The if structure in AVAP is a versatile tool that allows atomic comparisons or the evaluation of complex logical expressions processed by the dynamic evaluation engine.
Standard Interface
if(variable_A, value_B, operator)
Available Operators
Operator Description Example
= Strict equality (or numeric equivalence). if(role, "admin", "=")
!= Inequality. if(status, 200, "!=")
> / < Numeric magnitude comparison. if(age, 18, ">")
in Checks whether an element belongs to a list or string. if(user, blacklist, "in")
Complex Expression Evaluation
AVAP allows omission of comparison parameters to evaluate a complete logical expression directly in the third parameter.
Example:
if(None, None, "age >= 18 and balance > 100")
Block Closure Structure
An if block may include an optional else() block and must always terminate with the end() command.
3.2 Iterations and Loops (startLoop / endLoop)
For collection processing (such as database rows or parameter lists), AVAP implements an index-controlled loop structure.
Interface
startLoop(counter, start, end)
Execution Mechanics
Initialization:
The engine creates the counter variable with the start value.
Increment:
On each iteration, the counter automatically increases by 1.
Exit Condition:
The loop terminates when the counter exceeds the end value.
Practical Example: List Processing
// Retrieve the length of a list captured in Section II
getListLen(received_items, total)
startLoop(i, 0, total)
current_item = received_items[i]
// Processing logic for each item...
endLoop()
3.3 Error Handling and Robustness (try / exception)
AVAP is designed for production environments where external failures (database timeouts, third-party API outages) are expected realities. The try block allows capturing such events without stopping the server.
Interface
try ... exception(error_variable) ... end()
Technical Operation
try Block:
The engine attempts to execute the contained instructions. If a critical failure occurs, execution of that block stops immediately.
exception Block:
If an error is detected, control passes to this block. The error_variable is automatically populated with a string describing the failure (simplified stack trace).
end() Block:
Closes the structure and allows the script to continue normal execution after error handling.
Connector Safety Example
try
// Attempt a query to an external connector (Section V)
result = db.query("SELECT * FROM payments")
exception(failure_detail)
// If it fails, log the error and notify
addVar(_status, 500)
addVar(message, "Persistence error: %s" % failure_detail)
addResult(message)
end()
3.4 Early Exit Control (return)
The return() command is a control instruction that immediately terminates execution of the current context (either a function or the main script).
If used inside a function, it returns control (and optionally a value) to the caller.
If used in the main flow, it terminates API execution and triggers automatic delivery of the JSON response constructed up to that point.
Examples
1. Simple Comparison
Code snippet
addParam("lang", l)
if(l, "es", "=")
addVar(msg, "Hello")
end()
addResult(msg)
2. Standard Else
Code snippet
if(balance, 0, ">")
allow = True
else()
allow = False
end()
addResult(allow)
3. Complex Expression (Dynamic Evaluation)
Code snippet
if(None, None, "user_type == 'VIP' or purchases > 100")
addVar(discount, 0.20)
end()
addResult(discount)
4. Loop from 1 to 10 (ID Generation)
Code snippet
startLoop(i, 1, 10)
AddvariableToJSON("item_${i}", "generated_value", my_json)
endLoop()
addResult(my_json)
5. HTTP Request Try-Catch
Code snippet
try
RequestGet("https://api.test.com/data", 0, 0, response)
exception(e)
addVar(error_trace, "Connection failure: %s" % e)
addResult(error_trace)
end()
6. Proper Loop Exit (Using Control Variable)
Code snippet
found = False
startLoop(i, 1, 10)
if(i, 5, "==")
found = True
// In AVAP, to exit you can force the index beyond the limit
i = 11
end()
endLoop()
addResult(found)
7. 'in' Validation (Membership Check)
Code snippet
addParam("role", r)
if(r, ["admin", "editor", "root"], "in")
access = True
end()
addResult(access)
8. Loop Over Data Length
Code snippet
getListLen(records, total)
startLoop(idx, 0, total)
current = records[idx]
// processing logic
endLoop()
9. Inequality If
Code snippet
if(new_password, old_password, "!=")
addVar(change, "Password updated")
end()
addResult(change)
10. Critical SQL Error Handling
Code snippet
try
ormDirect("UPDATE nonexistent_table SET a=1", res)
exception(e)
addVar(_status, 500)
addResult("Database error")
end()

48
ingestion/docs/1_Introduction.txt Normal file
View File

@ -0,0 +1,48 @@
Introduction
Discovering a New Programming Language
Welcome to the AVAP book, where you will delve into the fascinating world of an innovative and powerful programming language: AVAP™. In these pages, we will explore together the fundamental concepts, syntax, and unique features of AVAP™, and prepare you to master this new language and harness its full potential in your software development projects.
Discovering AVAP
AVAP™ is much more than just a programming language; it is a versatile tool designed to enhance creativity and efficiency in software development. With its clear and expressive syntax, AVAP™ allows developers to write code more quickly and concisely, without sacrificing the power and flexibility needed to create robust and scalable applications.
What Makes AVAP Special?
AVAP™ stands out due to several distinctive features that make it unique in the programming world:
Integrated Virtualization: AVAP™ is designed from the ground up with the concept of virtualization in mind. Every aspect of the language is optimized to work in virtual environments, allowing developers to create immersive and scalable experiences.
Powerful APIs: AVAP™ provides a comprehensive set of tools for interacting with external APIs and web services, making it easier to integrate advanced functionalities into your applications.
Enhanced Productivity: With an intuitive syntax and advanced abstraction features, AVAP™ allows you to write less code to achieve more, thereby increasing your productivity and accelerating development time.
What Will You Find in This Book?
In this book, we will guide you through the basic and advanced concepts of AVAP™, providing practical examples, useful tips, and challenging exercises to help you master the language and become an expert AVAP™ developer. From installing and configuring the development environment to creating complete applications, this book will accompany you every step of the way towards mastering AVAP™.
Are You Ready to Get Started?
Then let’s not wait any longer! Dive into the pages of this book and get ready to embark on an exciting journey towards mastering AVAP™. Whether you are an experienced programmer looking for new tools or a curious beginner in the world of programming, this book has something for you. Let’s explore the fascinating world of AVAP™ together!
The Virtuality Attribute in AVAP™
AVAP™ (Advance Virtual API Programming) is a dynamic programming language distinguished by its virtuality attribute, which enables the development of virtual APIs in a dynamic and flexible manner. This attribute is based on the fact that the language specifications do not reside in the language interpreter, allowing the final code to be constructed in real-time by the language server.
1.1 Virtuality Principle in AVAP
The principle of virtuality in AVAP™ is based on several key aspects:
1.1.1 Language Specifications Decoupled from the Interpreter
In AVAP™, language specifications are not compiled into the core of the language nor do they reside in the interpreter. This means that the interpreter is not tied to a specific implementation of the language, providing great flexibility and adaptability in code interpretation.
1.1.2 Dynamic Code Construction in Real-Time
Thanks to the virtuality attribute, AVAP™ allows for dynamic code construction in real-time. This means that the final code to be interpreted by the language server can vary and mutate according to current needs, without the need for recompilation or redistribution.
1.1.3 Development of Dynamic Virtual APIs
The virtuality attribute in AVAP™ enables the development of virtual APIs in a dynamic manner. This allows APIs to evolve, improve, and adapt to new security or functional needs in real-time, without affecting the clients utilizing the API endpoint.
1.2 Benefits of the Virtuality Attribute
Flexibility: The ability to construct code in real-time provides significant flexibility in API development and management.
Agility: The capacity to adapt and evolve without the need for precompilation or distributed updates allows for greater agility in software development.
Simplified Maintenance: The development of dynamic virtual APIs simplifies the maintenance process, as changes do not need to be made to clients consuming those APIs.
1.3 Interaction with Artificial Intelligence
One of the most innovative features of this language is its integration with artificial intelligence through OpenAI. This integration allows the language to automatically generate the necessary results through an interface with OpenAI once the programmer has a clear solution to a problem. This functionality not only speeds up development but also reduces the margin of error and improves efficiency.
1.4 Access to Databases
The language also includes the capability to interact with databases using natural language, supported by artificial intelligence, currently version XXXXX through OpenAI. This feature allows for complex queries and data manipulation without deep knowledge of SQL, simplifying development and improving accessibility for programmers of all levels.
With this guide, we hope to provide you with all the necessary information to make the most of this dynamic language's capabilities. From variable management to automated result generation and simplified database access, this language is designed to transform the way you develop APIs.
1.5 Conclusions
The virtuality attribute in AVAP™ represents an innovative approach to virtual API development, allowing for greater flexibility, agility, and simplification in the software development and maintenance process. By decoupling language specifications from the interpreter and enabling dynamic code construction in real-time, AVAP™ offers a new paradigm in API design and management.

179
ingestion/docs/20_Concurrency_asynchrony.txt Normal file
View File

@ -0,0 +1,179 @@
SECTION IV: Concurrency and Asynchrony
AVAP implements a thread-based concurrency model that enables fire-and-forget execution or parallel execution with later synchronization. This is essential for tasks such as email dispatching, log processing, or querying multiple external APIs simultaneously.
4.1 Launching Background Processes (go_async)
The go_async command extracts a block of code from the main sequential flow and places it into a parallel execution queue.
Interface
go_async(thread_id)
Execution Mechanics
Identification:
The programmer assigns a thread_id (a string or variable) to reference the process later.
Forking:
When invoked, the AVAP engine creates a new native thread. The main flow immediately continues to the next instruction after the go_async block.
Context Isolation:
The asynchronous thread inherits a snapshot copy of the variable state at the moment of invocation, allowing it to operate safely without interfering with the main thread.
Example: Immediate Response with Long-Running Process
addParam("email", destination)
go_async("email_dispatch")
// This block takes 5 seconds, but the API does not wait
mail_service.send(destination, "Welcome to AVAP")
end()
addVar(msg, "Your email is being processed in the background")
addResult(msg)
// The client receives the response in milliseconds
4.2 Result Synchronization (gather)
When the main flow requires data generated by an asynchronous thread to proceed, the gather synchronization mechanism is used.
Interface
gather(thread_id, timeout)
Specifications
thread_id:
The identifier used in the go_async command.
timeout (Seconds):
Maximum time the main thread will wait. If the asynchronous thread does not finish within this period, AVAP raises an exception that can be caught (see Section III).
Technical Behavior
Controlled Blocking:
The main thread is suspended until the specified thread_id completes.
State Recovery:
Once synchronized, any variables modified within the asynchronous thread are merged back into the main thread’s context.
4.3 Optimized Parallel Execution (Fan-Out Pattern)
AVAP allows launching multiple threads and then waiting for all of them, reducing total execution time to the duration of the slowest thread instead of the sum of all execution times.
Example: Querying Multiple Databases
go_async("db_north")
data_north = north_connector.query("SELECT...")
end()
go_async("db_south")
data_south = south_connector.query("SELECT...")
end()
// Wait for both (Maximum 10 seconds)
gather("db_north", 10)
gather("db_south", 10)
// Combine results using Section I mechanisms
total_result = data_north + data_south
addResult(total_result)
4.4 Promise State in Output
As mentioned in Section II, if a variable that is still being processed in an asynchronous thread is passed to addResult, AVAP manages the response intelligently:
If the thread is still running:
The output JSON will display "variable": "promised" or the thread ID.
If the thread failed:
The error is logged internally and the variable is set to None.
If gather was used before addResult:
The fully processed real value is returned in the response.
Examples
1. Background Process Fire-and-Forget
Code snippet
go_async("notification_thread")
// This block runs on its own
RequestPost("https://hooks.slack.com/...", {}, body, r)
end()
2. Synchronization (Gather)
Code snippet
go_async("heavy_calc")
result = (500 * 250) / 3
end()
gather("heavy_calc", 10) // Waits until it finishes
addResult(result)
3. Connector Parallelism
Code snippet
go_async("db_1")
res1 = db1.query("SELECT...")
end()
go_async("db_2")
res2 = db2.query("SELECT...")
end()
4. Gather with Safety Timeout
Code snippet
gather("external_process", 5)
// If it doesn't finish in 5s, the script continues or fails depending on config
5. Using "Promised" State
Code snippet
go_async("long_task")
final = "Finished"
end()
addResult(final) // You will see "promised" in the JSON
6. Validation After Gather
Code snippet
gather("data_thread", 2)
if(thread_data, None, "!=")
addResult(thread_data)
end()
7. Multi-Threading for Auditing
Code snippet
go_async("audit_thread")
ormDirect("INSERT INTO audit...", r)
end()
8. Dynamic Gather in a Loop
Code snippet
startLoop(h, 1, 3)
gather("thread_${h}", 1)
endLoop()
9. Asynchronous Tax Calculation
Code snippet
go_async("tax_engine")
total_tax = total * 0.21
end()
10. Try-Except Inside an Async Thread
Code snippet
go_async("safe_thread")
try
// Task that may fail
exception(err)
// Log the error locally
end()
end()

175
ingestion/docs/21_Persistance_connectors_orm.txt Normal file
View File

@ -0,0 +1,175 @@
SECTION V: Persistence, Connectors, and Native ORM
AVAP is designed to be database-agnostic. It enables data manipulation through three layers: the universal connector, simplified ORM commands, and direct SQL execution.
5.1 The Universal Connector (avapConnector)
The avapConnector command is the entry point for any external integration. It uses a Connection Token system (Base64) that encapsulates configuration details (host, port, credentials, driver) to keep code clean and secure.
Interface
connector_variable = avapConnector("BASE64_TOKEN")
Connector Object Capabilities
Once instantiated, the variable behaves as an object with dynamic methods:
Database Connectors:
Expose the .query(sql_string) method, which returns objects or lists depending on the result set.
API Connectors (Twilio, Slack, etc.):
Expose native service methods (e.g., .send_sms()).
Example: Dynamic Assignment with Connectors
// Instantiate the connection
db = avapConnector("REJfQ09OTkVDVE9SM...")
// Execute query and use Section I dynamic evaluation
users = db.query("SELECT * FROM users")
first_admin = users[0].name if users[0].role == 'admin' else 'N/A'
addResult(first_admin)
5.2 Native ORM Layer (ormCheckTable / ormDirect)
For quick operations on the local or default database cluster, AVAP provides system-level commands that do not require prior instantiation.
5.2.1 ormCheckTable
Verifies the existence of a database structure. It is critical for installation scripts or automated migrations.
Interface:
ormCheckTable(table_name, target_var)
Response:
target_var receives the string values "True" or "False".
5.2.2 ormDirect
Executes SQL statements directly. Unlike .query(), it is optimized for statements that do not necessarily return rows (such as INSERT, UPDATE, or CREATE TABLE).
Interface:
ormDirect(statement, target_var)
Interpolation Usage Example:
ormDirect("UPDATE users SET login = '%s' WHERE id = %s" % (now, id), result)
5.3 Data Access Abstraction (Implicit Commands)
AVAP includes specialized commands for common CRUD operations, reducing the need to write manual SQL and mitigating injection risks.
ormAccessSelect
Performs filtered queries returning a list-of-objects structure.
Syntax:
ormAccessSelect(table, filters, target)
ormAccessInsert / ormAccessUpdate
Manages data persistence.
If used on an object that already has an ID, Update synchronizes changes; otherwise, Insert creates the record.
5.4 Dynamic Query Formatting (Injection Prevention)
As detailed in Section I, the AVAP engine processes SQL strings before sending them to the database engine. The official recommendation is to always use interpolation with the % operator to ensure proper handling of data types (Strings vs Integers) by the driver.
Recommended Secure Pattern
sql = "SELECT * FROM %s WHERE status = '%s'" % (table_name, recovered_status)
res = db.query(sql)
5.5 Cryptographic Security Integration (encodeSHA256)
Within the persistence flow, AVAP provides native tools to secure sensitive data before it is written to disk.
Interface
encodeSHA256(source_text, target_variable)
Complete Registration Flow (Final Example)
This example integrates Sections I, II, III, and V:
// II: Input capture
addParam("pass", p)
addParam("user", u)
// I & V: Processing and security
encodeSHA256(p, secure_pass)
// V: Insertion
sql = "INSERT INTO users (username, password) VALUES ('%s', '%s')" % (u, secure_pass)
ormDirect(sql, db_result)
// III & II: Response
if(db_result, "Success", "=")
addVar(msg, "User created")
addResult(msg)
end()
Examples
1. Connector Instantiation
Code snippet
my_db = avapConnector("VE9LRU5fREVCX0RFU0FSUk9MTE8=")
2. Record Retrieval
Code snippet
rows = my_db.query("SELECT id, name FROM users")
addResult(rows)
3. Direct Command Execution
Code snippet
ormDirect("TRUNCATE TABLE temp_cache", status)
4. Structure Verification
Code snippet
ormCheckTable("inventory", exists)
if(exists, "False", "==")
ormDirect("CREATE TABLE inventory...", r)
end()
5. Secure Update (Interpolation)
Code snippet
sql = "UPDATE users SET login_count = %s WHERE email = '%s'" % (count, email)
ormDirect(sql, res)
6. JSON/DB Object Navigation
Code snippet
found_id = query_result[0].id
addResult(found_id)
7. ORM Select with Filter
Code snippet
ormAccessSelect("orders", {"status": "pending"}, list_result)
addResult(list_result)
8. Processing Database Results
Code snippet
records = db.query("SELECT...")
startLoop(i, 0, len(records))
name = records[i].name
endLoop()
9. Cryptographic Persistence
Code snippet
encodeSHA256(password_raw, hashed)
ormDirect("INSERT INTO logins (hash) VALUES ('%s')" % hashed, r)
10. Third-Party Connector (e.g., Slack)
Code snippet
slack_api = avapConnector("U0xBQ0tfQVBJX1RPS0VO")

157
ingestion/docs/22_System_utilities_transformation.txt Normal file
View File

@ -0,0 +1,157 @@
SECTION VI: System Utilities and Transformation
This section documents the native commands for advanced string manipulation, precise time handling, and dynamic data generation.
6.1 Time and Date Management (getDateTime / stampToDatetime)
AVAP handles time in two formats:
Epoch/Timestamp (numeric): Ideal for calculations.
Formatted Datetime (string): Ideal for human readability and database storage.
6.1.1 getDateTime
Generates the current time with high precision.
Interface:
getDateTime(format, timeDelta, timeZone, targetVar)
Parameters
format:
Example: "%Y-%m-%d %H:%M:%S".
If left empty, returns the current Epoch timestamp.
timeDelta:
Seconds to add (positive) or subtract (negative).
Particularly useful for calculating token expiration times.
timeZone:
Time zone region (e.g., "Europe/Madrid").
6.1.2 stampToDatetime
Converts a numeric value (Unix Timestamp) into a human-readable string.
Interface:
stampToDatetime(timestamp, format, offset, targetVar)
Common Use Case:
Formatting dates retrieved from the database (Section V) before sending them to the client (Section II).
6.2 Advanced String Manipulation (replace / randomString)
6.2.1 replace
Allows text cleaning and transformation. Essential when receiving client data that requires sanitization.
Interface:
replace(sourceText, oldText, newText, targetVar)
Example Use Case:
Removing spaces or unwanted characters from a username before executing a SQL query.
6.2.2 randomString
Generates secure random alphanumeric strings.
Interface:
randomString(length, targetVar)
Applications:
Temporary password generation
Session ID creation
Unique file name generation
6.3 Security and Hash Operations (encodeSHA256)
Although previously mentioned in the persistence section, this is fundamentally a data transformation utility.
Mechanics
Deterministic one-way function.
AVAP uses an optimized implementation ensuring that the same input always produces the same hash.
This enables secure login comparisons without storing or exposing the actual password.
6.4 The Return Command (return)
Within functions and execution flows, return not only stops execution but can also inject the result of a subroutine back into the main flow.
Complete Utility Flow Example
// 1. Generate a temporary token
randomString(16, token_raw)
// 2. Calculate expiration (within 1 hour = 3600 seconds)
getDateTime("%Y-%m-%d %H:%M:%S", 3600, "UTC", expiration_date)
// 3. Format a system message using Section I
message = "Your token %s expires on %s" % (token_raw, expiration_date)
// 4. Send to client (Section II)
addResult(message)
6.5 Common Format Tokens (Cheat Sheet)
Token Description Example
%Y Full year 2026
%m Month (01–12) 02
%d Day (01–31) 23
%H Hour (00–23) 21
%M Minute (00–59) 45
Examples
1. Unix Timestamp Retrieval
Code snippet
getDateTime("", 0, "UTC", now)
addResult(now)
2. Database-Formatted Date
Code snippet
getDateTime("%Y-%m-%d %H:%M:%S", 0, "Europe/Madrid", sql_date)
addResult(sql_date)
3. Expiration Calculation (1 Day)
Code snippet
getDateTime("", 86400, "UTC", expires_at)
addResult(expires_at)
4. Timestamp to Readable Conversion
Code snippet
stampToDatetime(1708726162, "%d/%m/%Y", 0, human_date)
addResult(human_date)
5. String Cleaning (Replace)
Code snippet
replace("REF_1234_OLD", "OLD", "NEW", updated_ref)
addResult(updated_ref)
6. Random Token Generator
Code snippet
randomString(32, security_token)
addResult(security_token)
7. SHA256 Hash for Integrity
Code snippet
encodeSHA256("payload_data", checksum)
addResult(checksum)

98
ingestion/docs/23_Function_architecture_scopes.txt Normal file
View File

@ -0,0 +1,98 @@
SECTION VII: Function Architecture and Scopes
This section explains how to encapsulate reusable logic and how AVAP manages isolated memory to prevent side effects across different parts of the program.
7.1 Definition and Declaration (function)
A function in AVAP is an independent block of code registered in the engine so it can be invoked at any time.
Interface
function function_name(argument1, argument2, ...){ ... }
Technical Characteristics
Local Scope (function_local_vars):
When entering a function, AVAP creates a new local variable dictionary. Variables created inside the function (e.g., temp = 10) do not exist outside of it, protecting the global state.
Context Inheritance:
Functions can read global variables using the $ prefix, but any new assignment (=) remains in the local scope unless an explicit global persistence command is used.
7.2 The Return Command (return)
This is the mechanism used to terminate function execution and optionally send a value back to the caller.
Interface
return(variable_or_value)
Behavior
Termination:
Immediately stops function processing.
Data Transfer:
The value passed to return is injected into the variable that invoked the function in the main flow.
Cleanup:
Once return executes, the function’s local variable dictionary is destroyed to free memory.
7.3 Invocation and Parameter Passing
Functions are called by name followed by the required values or variables.
Professional Implementation Example
// Function definition (Local Scope)
function calculate_discount(base_price, percentage){
factor = percentage / 100
discount = base_price * factor
total = base_price - discount
return(total)
}
// Main Flow (Global Scope)
addVar(pvp, 150)
// Function call passing a reference $ and a literal value
final_price = calculate_discount($pvp, 20)
addResult(final_price) // Result: 120
7.4 Functions as Middlewares
In the registerEndpoint command (Section I), the middleware parameter accepts a list of functions. These functions have special behavior:
If a middleware executes a return() without a value or with an error value, AVAP can be configured to abort the request before reaching the main handler.
Ideal for guard tasks such as:
API key verification
Data schema validation
Initial audit logging
7.5 Recursion and Limits
AVAP supports recursion (a function calling itself), but caution is recommended regarding stack depth, especially in asynchronous processes (Section IV).
For processing large volumes of data, it is always preferable to use startLoop (Section III) instead of deep recursive calls.
Examples
1. Modular Sum Function
Code snippet
function sum(a, b){
total = a + b
return(total)
}
result = sum(10, 20)
2. Access Validation Function
Code snippet
function is_valid(token){
if(token, "SECRET", "==")
return(True)
end()
return(False)
}
authorized = is_valid($input_token)

33
ingestion/docs/24_Master_example.txt Normal file
View File

@ -0,0 +1,33 @@
Master Example (Combining Sections)
This example shows how a real flow uses almost all sections:
// SECTION I & II: Registration and Input
registerEndpoint("/v1/user", "POST", [], "Create User", main, final_res)
function main(){
addParam("user", u)
addParam("pass", p)
// SECTION III & VI: Validation and Security
if(u, None, "==")
addVar(_status, 400)
return("User is required")
end()
encodeSHA256(p, pass_hash)
// SECTION IV: Asynchrony (Audit log)
go_async("audit")
ormDirect("INSERT INTO audit (event) VALUES ('User creation attempt')", r)
end()
// SECTION V: Persistence
db = avapConnector("TOKEN_DB")
res_db = db.query("INSERT INTO users (name, pass) VALUES ('%s', '%s')" % (u, pass_hash))
// SECTION II: Response
addVar(_status, 201)
addResult(res_db)
return(res_db)
}

48
ingestion/docs/2_Dynamic_Programming_Language.txt Normal file
View File

@ -0,0 +1,48 @@
Chapter 1: Dynamic Programming Language
In this chapter, we will introduce AVAP™ as a dynamic programming language. A dynamic language is one whose behavior can be modified during the runtime of the program. AVAP™ shares many characteristics with other dynamic languages, making it a powerful and versatile tool for application development.
1.1 Features of AVAP™ as a Dynamic Language
AVAP™ is characterized by its dynamic nature, which means it offers various features that allow flexibility and adaptability in program development. Below, we will detail some of these features:
1.1.1 Dynamic Typing
In AVAP™, variable typing is dynamic, which means it is not necessary to explicitly declare the type of a variable before assigning it a value. This allows greater flexibility in data handling and simplifies code writing.
# Example of dynamic typing
x = 10 # x is an integer
x = "Hello" # x is now a string
1.1.2 Automatic Memory Management
AVAP™ uses an automatic garbage collector to manage memory dynamically. This means that developers do not have to worry about manually allocating and freeing memory, which simplifies the development process and reduces the likelihood of memory management-related errors.
# Example of automatic memory management:
list = [1, 2, 3, 4, 5]
# There is no need to free the memory of the list after use
1.1.3 Runtime Interpreter: Dynamic Code Construction
AVAP™ uses a runtime interpreter that goes beyond simply executing code line by line. Instead, the AVAP™ runtime interpreter is characterized by its ability to dynamically construct code during runtime, adding an element of virtuality to the execution process.
Dynamic code construction means that the AVAP™ runtime interpreter can generate and modify code as the program executes. This allows for greater flexibility and adaptability in data manipulation and operation execution.
A fundamental aspect of virtuality in dynamic code construction is that the language specifications are completely isolated from the runtime interpreter. This means that the interpreter is not tied to a specific language implementation, facilitating code portability and allowing for the transparent integration of new features and functionalities.
In summary, the AVAP™ runtime interpreter not only executes code line by line but also dynamically constructs code during runtime, adding an additional level of virtuality and flexibility to the program execution process.
1.1.4 Flexibility in Programming
AVAP™ offers a wide range of features that promote flexibility in programming. This includes support for higher-order functions, dynamic exception handling, and the ability to manipulate objects at runtime, among others.
# Example of a higher-order function
def operation(func, a, b):
return func(a, b)
def add(x, y):
return x + y
result = operation(add, 3, 5)
# The add function is passed as an argument
1.2 Advantages of AVAP™ as a Dynamic Language
As a dynamic programming language, AVAP™ offers several advantages, including:
Greater flexibility and adaptability in program development.
Faster writing and execution of code.
Facilitates experimentation and exploration of solutions.
Allows for rapid feedback during development.
1.3 Summary
AVAP™ is a dynamic programming language that offers a wide range of features promoting flexibility, adaptability, and speed in application development. With its dynamic typing, automatic memory management, runtime interpreter, and programming flexibility, AVAP™ becomes a powerful and versatile tool for developers.

78
ingestion/docs/3_Notation.txt Normal file
View File

@ -0,0 +1,78 @@
Chapter 2: Notation in AVAP™
Introduction
Notation in AVAP™ refers to the conventions and rules used to write and format code in the AVAP™ programming language. Notation is essential to ensure code readability and comprehension, as well as to establish a coherent and consistent syntax across all projects.
General Conventions
In AVAP™, several general notation conventions are followed, similar to those used in other programming languages like Python. Some of these conventions include:
Indentation: Code is structured through indentation, using white spaces or tabs to indicate the hierarchy and structure of the code. It is recommended to use four spaces for each level of indentation.
Case Sensitivity: AVAP™ is case-sensitive, meaning that identifiers, variable names, and keywords must be consistently written using the same capitalization format throughout the code.
Comments: Comments are used to document the code and explain its functionality. Single-line comments begin with the // symbol, while multi-line comments start with /* and end with */.
Specific Notation Rules
In addition to general conventions, AVAP™ follows specific notation rules for different elements of the language, including:
Variables: Variable names should be descriptive and meaningful, using lowercase letters and underscores to separate words if necessary for readability (e.g., variable_name).
Functions: Function names should follow the same conventions as variables, with the addition of parentheses to indicate function parameters (e.g., function_name(parameter1, parameter2)).
Constants: Constants are typically written in uppercase letters with underscores separating words (e.g., EXAMPLE_CONSTANT).
The descriptions of lexical analysis and syntax use a modified Backus–Naur form (BNF) grammar notation. This uses the following style of definition:
<program> ::= <statement_list>
<statement_list> ::= <statement> | <statement> <statement_list>
<statement> ::= <global_assignment> | <local_assignment> | <command>
<global_assignment> ::= "addVar(" <string_value> "," <variable_name> ")"
<local_assignment> ::= <variable_name> "=" <value>
<string_value> ::= """ <string_content> """
<string_content> ::= <string_part> | <string_part> <string_content>
<string_part> ::= <text> | <variable_reference>
<text> ::= <character> | <character> <text>
<variable_reference> ::= " <variable_name> "
<variable_name> ::= <letter> | <letter> <variable_name>
<value> ::= <string_value> | <number> | <expression>
<number> ::= <digit> | <digit> <number>
<expression> ::= <value> | <value> <operator> <value>
<operator> ::= "+" | "-" | "*" | "/"
<command> ::= <any_valid_command_syntax>
<character> ::= any character except `" ` and `\`
<letter> ::= "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" | "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z" | "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" | "_"
<digit> ::= "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
Explanation:
<program>: A program is a list of statements.
<statement_list>: A list of statements can be a single statement or a statement followed by another list of statements.
<statement>: A statement can be a global assignment, a local assignment, or a command.
<global_assignment>: A global assignment follows the format addVar('value', variable_name).
<local_assignment>: A local assignment follows the Python syntax variable_name = value.
<string_value>: A string value is enclosed in double quotes and contains string content.
<string_content>: The content of a string can be a string part or a string part followed by more string content.
<string_part>: A string part can be literal text or a variable reference.
<text>: Text is a series of characters.
<variable_reference>: A variable reference follows the format $ variable .
<variable_name>: A variable name can be a letter or a combination of letters.
<value>: A value can be a string value, a number, or an expression.
<number>: A number can be a digit or a series of digits.
<expression>: An expression can be a value or a combination of two values with an operator.
<operator>: An operator can be +, -, *, or /.
<command>: A command can be any valid command syntax.
<character>: A character can be any character except double quotes and the backslash.
<letter>: A letter can be an alphabetical character, a digit, or an underscore.
<digit>: A digit is a number from 0 to 9.
This BNF notation covers the assignment of global and local variables, as well as variable substitution in strings.
Practical Example
// Definition of a variable
example_variable = 10
// Definition of a function
example_function(parameter):
// Function body
result = parameter * 2
return result
// Function call
result = example_function(example_variable)
In this example, notation conventions are used to define a variable, a function, and to call the function with a parameter.
Conclusions
Notation in AVAP™ is a fundamental part of software development in the language. By following clear and consistent notation conventions, developers can write and maintain code more effectively, contributing to the readability, understanding, and maintainability of the code in projects of any size and complexity.
With this understanding of notation in AVAP™, developers can write clean and structured code that is easy to understand and maintain over time.

74
ingestion/docs/4_Lexics.txt Normal file
View File

@ -0,0 +1,74 @@
Introduction
Lexical analysis is the first step in the process of compiling or interpreting a program in AVAP™. It involves breaking down the source code into lexical components or "tokens," which are the smallest units of meaning in the language. These tokens include keywords, identifiers, operators, punctuation symbols, and literals.
Lexical Components in AVAP™
The lexical components in AVAP™ are similar to those in other programming languages like Python. Some of the most common lexical components in AVAP™ include:
Keywords: These are reserved words that have a special meaning in the language and cannot be used as variable or function names. Examples of keywords in AVAP™ include if, else, for, while, return, among others.
Identifiers: These are names given to variables, functions, and other elements of the program by the programmer. Identifiers must follow certain formatting rules and cannot match keywords. For example, variable, example_function, result are examples of identifiers in AVAP™.
Operators: These are symbols used to perform operations in the program. Examples of operators in AVAP™ include +, -, *, /, =, ==, !=, among others.
Literals: These represent constant values in the program, such as integers, floating-point numbers, text strings, and boolean values. Examples of literals in AVAP™ include 10, 3.14, "text", True, False, among others.
Punctuation Symbols: These are special characters used to separate elements of the code and define the structure of the program. Examples of punctuation symbols in AVAP™ include (), , [], ,, :, ;, among others.
Lexical Analysis Process
The lexical analysis process in AVAP™ consists of several steps:
Scanning: The source code is read sequentially, and the lexical components are identified. Regular expressions are used to recognize patterns corresponding to keywords, identifiers, operators, etc.
Tokenization: The identified lexical components are converted into tokens, which are objects representing each component with its associated type and value.
Token Generation: The generated tokens are passed to the next step of the compilation or interpretation process for syntactic and semantic analysis.
Keywords
Keywords in AVAP are reserved words that have specific meanings and cannot be used as identifiers. The keywords in AVAP are:
randomString
ormAI
functionAI
stampToDatetime
getTimeStamp
getRegex
getDateTime
encodeMD5
encodeSHA256
getQueryParamList
getListLen
ormCheckTable
ormCreateTable
end
else
if
endLoop
startLoop
ormAccessInsert
ormAccessSelect
variableToList
RequestPost
RequestGet
addResult
AddvariableToJSON
addParam
variableFromJSON
itemFromList
addVar
function
return
Practical Example
Below is a practical example that illustrates lexical analysis in AVAP™:
// Function definition
function_example(parameter):
result = parameter * 2
return result
// Function call
value = function_example(10)
In this example, the lexical analysis would identify the following tokens:
function_example: Function identifier.
(, ): Punctuation symbols.
parameter, result, value: Variable identifiers.
=, *, 2: Operators.
10: Integer literal.
Conclusions
Lexical analysis is a crucial step in the compilation or interpretation of a program in AVAP™. By breaking down the source code into tokens, it lays the foundation for subsequent syntactic and semantic analysis, allowing the program to be correctly understood and executed by the interpreter or compiler.
With a clear understanding of lexical analysis in AVAP™, developers can write clean and structured code, facilitating the software development process in the language.

48
ingestion/docs/5_Data_Model.txt Normal file
View File

@ -0,0 +1,48 @@
Introduction
The data model in AVAP™ defines how data is organized and manipulated within the language. Similar to Python, AVAP™ uses a flexible and dynamic data model that allows for working with a wide variety of data types and data structures.
Data Types
In AVAP™, just like in Python, data types are categories that represent different kinds of values that can be stored and manipulated in a program. Some of the most common data types in AVAP™ include:
Integers (int): Represent whole numbers, positive or negative, without a fractional part.
Floating-point numbers (float): Represent numbers with both integer and fractional parts.
Strings (str): Represent sequences of Unicode characters.
Booleans (bool): Represent truth values, either True or False.
Lists (list): Ordered and mutable collections of elements.
Tuples (tuple): Ordered and immutable collections of elements.
Dictionaries (dict): Unordered collections of key-value pairs.
Sets (set): Unordered collections of unique elements.
Data Structures
In addition to individual data types, AVAP™ provides various data structures that allow for more complex organization and manipulation of data:
Lists: Created using square brackets [ ] and can contain any data type, including other lists.
Tuples: Created using parentheses ( ) and are immutable, meaning they cannot be modified once created.
Dictionaries: Created using curly braces and store key-value pairs, where each key is unique within the dictionary.
Sets: Created using curly braces and contain unique elements, meaning there are no duplicates in a set.
Data Structures
In addition to individual data types, AVAP™ provides various data structures that allow for more complex organization and manipulation of data:
Lists: Created using square brackets [ ] and can contain any data type, including other lists.
Tuples: Created using parentheses ( ) and are immutable, meaning they cannot be modified once created.
Dictionaries: Created using curly braces and store key-value pairs, where each key is unique within the dictionary.
Sets: Created using curly braces and contain unique elements, meaning there are no duplicates in a set.
Practical Example
Below is a practical example that illustrates the use of the data model in AVAP™:
# Definition of a list
example_list = [1, 2, 3, 4, 5]
# Accessing individual elements
print(example_list[0]) # Output: 1
# Slicing to get a sublist
sublist = example_list[2:4]
print(sublist) # Output: [3, 4]
# List methods
example_list.append(6)
print(example_list) # Output: [1, 2, 3, 4, 5, 6]
Conclusions
The data model in AVAP™ provides a flexible and dynamic structure for working with data in the language. By understanding the available data types, data structures, operations, and methods, developers can write efficient and effective code that manipulates and processes data effectively.

62
ingestion/docs/6_Data_Types.txt Normal file
View File

@ -0,0 +1,62 @@
Chapter 5: Data Types
In this chapter, we will explore the data types available in AVAP™. Data types are fundamental in programming as they determine what kind of values can be stored in a variable and what operations can be performed with those values. Throughout this chapter, we will discuss the basic data types in AVAP™ and how they are used in program development.
1.1 Basic Data Types
In AVAP™, like in Python, there are several basic data types:
1.1.1 Integers (int)
Integers represent whole numbers without decimals. They can be positive, negative, or zero. In AVAP™, integers are defined using the int data type.
integer_number = 10
1.1.2 Floating-Point Numbers (float)
Floating-point numbers represent real numbers with decimals. In AVAP™, they are defined using the float data type.
floating_number = 3.14
1.1.3 Strings (str)
Strings represent text. In AVAP™, they are defined using the str data type.
text_string = "Hello, world!"
1.1.4 Booleans (bool)
Booleans represent truth or falsehood values. In AVAP™, they are defined using the bool data type.
true_value = True
false_value = False
1.2 Conversion Between Data Types
In AVAP™, just like in Python, it is possible to convert between different data types using specific functions. Some common examples include:
1.2.1 Conversion to Integer
To convert a value to an integer, the int() function is used.
text = "10"
number = int(text)
1.2.2 Conversion to Floating-Point
To convert a value to a floating-point number, the float() function is used.
text = "3.14"
number = float(text)
1.2.3 Conversion to String
To convert a value to a string, the str() function is used.
number = 10
text = str(number)
1.3 Operations with Data Types
In AVAP™, just like in Python, it is possible to perform operations with different data types. For example:
# Operations with integers
a = 10
b = 5
sum = a + b
difference = a - b
# Operations with floating-point numbers
c = 3.5
d = 2.0
product = c * d
division = c / d
# Operations with strings
text1 = "Hello"
text2 = "world"
concatenation = text1 + " " + text2
1.4 Summary
Data types in AVAP™ are fundamental for program development. They allow for the storage and manipulation of different types of values, such as numbers, text, and truth values. With a solid understanding of data types and how they are used in program development, developers can create robust and functional applications in AVAP™.

44
ingestion/docs/7_Working_With_Variables.txt Normal file
View File

@ -0,0 +1,44 @@
Working with Variables
In this chapter, we will explore in detail working with variables in AVAP™. Variables are fundamental elements in programming as they allow us to store and manipulate data within a program. Throughout this chapter, we will examine the importance of variables, the types of local and global variables, as well as the different ways to declare them in AVAP™.
2.1 Importance of Variables
Variables play a crucial role in programming, as they allow us to store and manipulate data during the execution of a program. They enable the storage of temporary or permanent values, perform calculations, and facilitate communication between different parts of the program.
2.2 Types of Variables in AVAP™
In AVAP™, there are two main types of variables: local and global.
2.2.1 Local Variables
Local variables are those that are declared within a function or block of code and are only available within that scope. They have a limited scope, and their lifespan is restricted to the execution time of the block in which they are declared. Local variables are used to store temporary or intermediate data needed to perform calculations or execute operations within a function.
2.2.2 Global Variables
Global variables are those that are declared outside of any function or block of code and are available throughout the entire program. They have a global scope, and their lifespan lasts for the full duration of the program's execution. Global variables are used to store data that needs to be accessible from multiple parts of the program or that needs to retain its value over time.
2.3 Declaration of Variables in AVAP™
In AVAP™, variables can be declared in several ways:
2.3.1 addVar() Function
The addVar() function is used to declare global variables within the scope of an API. Its syntax is as follows:
addVar(variable_name, value)
Where:
variable_name is the name of the variable to be declared.
value is the initial value to be assigned to the variable (optional).
2.3.2 Direct Declaration
Local and global variables can also be declared directly without using the global statement, simply by assigning a value:
variable_name = value
Where:
variable_name is the name of the variable to be declared.
value is the initial value to be assigned to the variable.
2.3.3 Direct Initialization
It is also possible to declare and initialize a global variable at the same time using the following syntax:
global variable_name=value
Where:
variable_name is the name of the variable to be declared.
value is the initial value to be assigned to the variable, which automatically defines the variable's type.
2.4 Summary
Working with variables in AVAP™ is essential for developing efficient and scalable applications. Variables allow for storing and manipulating data during program execution, which facilitates calculations and communication between different parts of the program. With a solid understanding of variable types and the different ways to declare them in AVAP™, developers can create robust and functional applications.

40
ingestion/docs/8_How_to_work_with_comments.txt Normal file
View File

@ -0,0 +1,40 @@
How to Work with Comments
Comments are a fundamental tool in any programming language, as they allow you to document code, make it easier to understand, and help keep it organized. In the AVAP™ programming language, comments are an integral part of the syntax and are used to add additional information to the source code without affecting its execution.
Comments serve several purposes:
Documentation: Comments can be used to explain what specific parts of the code do, which can be helpful for anyone reading or maintaining the code in the future.
Clarification: They can clarify complex sections of code, making it easier for others (or yourself) to understand the logic and flow of the program.
Organization: Comments can help organize code by separating different sections or explaining the purpose of various code blocks.
Debugging: Comments can temporarily disable parts of code during debugging without deleting it, allowing you to test different scenarios.
In AVAP™, you can use different types of comments to suit your needs. They can be single-line comments or multi-line comments, depending on the level of detail and context required.
By incorporating comments into your code, you make it more maintainable and easier for others to follow, which is essential for collaborative projects and long-term code management.
3.1 Line Comments
Line comments in AVAP™ are used to add brief annotations or explanations to a specific line of code. These comments begin with the // symbol and continue until the end of the line. Everything following // is considered a comment and is ignored by the compiler.
// This is a line comment in AVAP™ int x = 5; // You can also add comments at the end of a line of code
Line comments are useful for providing quick clarifications about the code and improving its readability.
3.2 Block Comments
Block comments in AVAP™ are used to add comments that span multiple lines of code. These comments begin with /* and end with */. Everything between /* and */ is considered a comment and is ignored by the compiler.
/* This is a block comment in AVAP™ that spans multiple lines of code. It is used to explain extensive sections of code or to temporarily disable entire blocks of code. */
Block comments are ideal for providing detailed explanations about complex sections of code or for temporarily disabling entire blocks of code during debugging.
3.3 Documentation Comments
AVAP™ also supports documentation comments, which are used to automatically generate documentation from the source code. These comments begin with /// and are used to describe the functionality of classes, methods, variables, and other elements of the source code.
/// This function adds two integers and returns the result. /// \param a The first integer. /// \param b The second integer. /// \return The sum of the two numbers. int sum(int a, int b) return a + b;
Documentation comments are essential for maintaining up-to-date and detailed documentation of the code, which facilitates its understanding and use by other developers.
3.4 Best Practices
When using comments in AVAP™, it is important to follow some best practices:
Use comments moderately and only when necessary to clarify the code.
Keep comments updated as the code evolves.
Write clear and concise comments that are easy for other developers to understand.
Avoid redundant or unnecessary comments that do not provide useful information to the reader.
3.5 Summary
Comments in AVAP™ are an essential tool for improving the readability and maintainability of source code. With line comments, block comments, and documentation comments, developers can add explanations, clarifications, and useful documentation to the code, making it easier to understand and collaborate within development teams.

53
ingestion/docs/9_Expressions_in_avap.txt Normal file
View File

@ -0,0 +1,53 @@
Expressions in AVAP™
Introduction
Expressions in AVAP™ are combinations of values, variables, operators, and function calls that can be evaluated to produce a result. Just like in Python, expressions in AVAP™ can be simple or complex, and they can contain a variety of elements that manipulate and process data.
Types of Expressions
In AVAP™, as in Python, there are several types of expressions that can be used to perform different operations and calculations. Some of the most common types of expressions include:
Arithmetic: Perform mathematical operations such as addition, subtraction, multiplication, and division.
Logical: Evaluate logical conditions and return boolean values, such as True or False.
Comparative: Compare two values and return a result based on their relationship, such as equality, inequality, greater than, less than, etc.
Assignment: Assign a value to a variable.
Function Calls: Invoke functions and methods to perform specific tasks.
Operators
In AVAP™, as in Python, expressions can include a variety of operators that perform specific operations on data. Some of the most common operators include:
Arithmetic: +, -, *, /, %, etc.
Logical: and, or, not.
Comparative: ==, !=, >, <, >=, <=, etc.
Assignment: =, +=, -=, *=, /=, etc.
Working with Lists
Lists are a very versatile data structure in AVAP™ that allows you to store collections of elements of different types. Expressions in AVAP™ can involve operations and manipulations of lists, such as accessing individual elements, concatenation, searching, deletion, and more.
// Definition of a list
my_list = [1, 2, 3, 4, 5]
// Accessing individual elements
first_element = my_list[0] // Output: 1
// Concatenation of lists
another_list = [6, 7, 8]
combined_list = my_list + another_list // Output: [1, 2, 3, 4, 5, 6, 7, 8]
// Length of a list
length = len(my_list) // Output: 5
// Searching in a list
is_present = 5 in my_list // Output: True
// Removing elements
my_list.remove(3) // Removes the element 3 from the list
Practical Example
Below is a practical example that illustrates the use of expressions in AVAP™ with lists:
// Definition of a list of numbers
numbers = [1, 2, 3, 4, 5]
// Calculation of the sum of the elements
total = sum(numbers) // Output: 15
// Checking if a number is present in the list
is_present = 6 in numbers // Output: False
Conclusions
Expressions in AVAP™ are a fundamental part of programming, allowing for a wide variety of data operations and manipulations. By understanding the different types of expressions and operators, as well as working with data structures such as lists, developers can write clear and effective code that meets the program's requirements.

81
notebooks/langgraph_agent_simple.ipynb
View File

File diff suppressed because one or more lines are too long

1
pyproject.toml
View File

@ -27,6 +27,7 @@ dependencies = [
dev = [ dev = [
"accelerate>=1.12.0", "accelerate>=1.12.0",
"beir>=2.2.0", "beir>=2.2.0",
"boto3>=1.42.58",
"evidently>=0.7.20", "evidently>=0.7.20",
"jupyter>=1.1.1", "jupyter>=1.1.1",
"langfuse<=3", "langfuse<=3",

4
scratches/acano/es_ingestion.ipynb
View File

@ -325,7 +325,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 11, "execution_count": 2,
"id": "09ce3e29", "id": "09ce3e29",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@ -340,7 +340,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 12, "execution_count": 4,
"id": "d575c386", "id": "d575c386",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],

4
scratches/acano/evaluate_retrieve.ipynb
View File

@ -326,7 +326,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 39, "execution_count": null,
"id": "ab1932b7", "id": "ab1932b7",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
@ -521,7 +521,7 @@
"synth = SingleHopSpecificQuerySynthesizer(llm=LangchainLLMWrapper(llm))\n", "synth = SingleHopSpecificQuerySynthesizer(llm=LangchainLLMWrapper(llm))\n",
"\n", "\n",
"generator = TestsetGenerator(llm=LangchainLLMWrapper(llm), embedding_model=LangchainEmbeddingsWrapper(embeddings))\n", "generator = TestsetGenerator(llm=LangchainLLMWrapper(llm), embedding_model=LangchainEmbeddingsWrapper(embeddings))\n",
"synthetic_dataset = generator.generate_with_chunks(\n", "synthetic_dataset = generator.generate_with_langchain_docs(\n",
" chunks=docs,\n", " chunks=docs,\n",
" testset_size=100,\n", " testset_size=100,\n",
" query_distribution=[(synth, 1.0)]\n", " query_distribution=[(synth, 1.0)]\n",

312
scratches/acano/generate_synthethic_data.ipynb Normal file
View File

@ -0,0 +1,312 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "b657efd2",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from datasets import load_dataset\n",
"\n",
"import boto3\n",
"from botocore.config import Config\n",
"from langchain_core.messages import SystemMessage, HumanMessage\n",
"\n",
"from src.utils.llm_factory import create_chat_model\n",
"from src.config import RAW_DIR, INTERIM_DIR"
]
},
{
"cell_type": "markdown",
"id": "e6e90339",
"metadata": {},
"source": [
"### Create llm instance"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "20eecc53",
"metadata": {},
"outputs": [],
"source": [
"config = Config(\n",
" region_name=\"us-east-1\",\n",
" connect_timeout=10, \n",
" read_timeout=600, \n",
")\n",
"\n",
"client = boto3.client(\"bedrock-runtime\", config=config)\n",
"\n",
"llm = create_chat_model(\n",
" provider=\"bedrock\",\n",
" client=client,\n",
" model=\"global.anthropic.claude-sonnet-4-6\",\n",
" temperature=0,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "96f12a22",
"metadata": {},
"source": [
"### Load mbpp data "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "78e29dc2",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],\n",
" num_rows: 374\n",
" })\n",
" test: Dataset({\n",
" features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],\n",
" num_rows: 500\n",
" })\n",
" validation: Dataset({\n",
" features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],\n",
" num_rows: 90\n",
" })\n",
" prompt: Dataset({\n",
" features: ['task_id', 'text', 'code', 'test_list', 'test_setup_code', 'challenge_test_list'],\n",
" num_rows: 10\n",
" })\n",
"})"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset_full = load_dataset(\"mbpp\")\n",
"dataset_full"
]
},
{
"cell_type": "markdown",
"id": "3e7544bb",
"metadata": {},
"source": [
"### Load AVAP data"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e88b2d3",
"metadata": {},
"outputs": [],
"source": [
"with open(RAW_DIR / \"avap.txt\", \"r\") as f:\n",
" avap_docs = f.read()"
]
},
{
"cell_type": "markdown",
"id": "c039d79f",
"metadata": {},
"source": [
"### Random sample mbpp"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b186af30",
"metadata": {},
"outputs": [],
"source": [
"# Sample 100 random examples from the test split\n",
"random_test_samples = dataset_full[\"test\"].shuffle(seed=42).select(range(50))\n",
"\n",
"# Save text and code in a dictionary\n",
"test_samples_dict = {\n",
" str(i): {\n",
" \"text\": sample[\"text\"],\n",
" \"code\": sample[\"code\"],\n",
" }\n",
" for i, sample in enumerate(random_test_samples)\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "21a0422d",
"metadata": {},
"source": [
"### Prompts"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c075e29",
"metadata": {},
"outputs": [],
"source": [
"PROMPT_MBPP = SystemMessage(\n",
" content=f\"\"\"\n",
" You are an AI agent responsible for generating a synthetic dataset for a programming language called AVAP (Advanced Virtual API Programming).\n",
"\n",
" You will receive a JSON object containing programming problems and their solutions written in Python.\n",
"\n",
" Your task is to:\n",
" 1. Convert each Python programming problem into an equivalent problem framed for the AVAP language.\n",
" 2. Replace the Python solution with a correct implementation written in AVAP.\n",
"\n",
" Requirements:\n",
" - The AVAP solution must follow AVAP syntax.\n",
" - Use AVAP libraries and constructs when appropriate.\n",
" - Functions in AVAP are declared using the `function` keyword.\n",
" - Preserve the original problem intent and difficulty.\n",
" - Do not produce Python code in the final solution.\n",
"\n",
" Documentation usage rules:\n",
" - You MUST rely exclusively on the AVAP documentation provided below.\n",
" - Do not invent syntax, functions, or libraries that are not described in the documentation.\n",
" - If the documentation does not contain enough information to solve the problem, return exactly:\n",
" \"I don't know how to answer this question based on the provided documentation.\"\n",
"\n",
" AVAP Documentation:\n",
" {avap_docs}\n",
" \"\"\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cbb459f8",
"metadata": {},
"outputs": [],
"source": [
"PROMPT_NO_MBPP = HumanMessage(\n",
" content=\"\"\"\n",
" I gave you all the context for a new programming language called AVAP (Advanced Virtual API Programming). \n",
" With that context, I'd like you to generate a \"golden dataset\" of 100 question and answer where the user query is asking code snippets and the response is the full code. \n",
" I want the questions and answers to cover the entire context provided, and I want there to be questions and answers at all possible levels, from questions with simple code snippets to more complex questions with more complex code. \n",
" \n",
" You must return a .json file with user_query and response for each question and answer\n",
" \"\"\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "ea1e824e",
"metadata": {},
"source": [
"### Generate dataset using mbpp dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8b7dcf2f",
"metadata": {},
"outputs": [],
"source": [
"llm_response = llm.invoke([PROMPT_MBPP, HumanMessage(content=str(test_samples_dict))])\n",
"print(llm_response.content)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "c6f5872e",
"metadata": {},
"outputs": [],
"source": [
"json_str = llm_response.content.removeprefix(\"```json\").removesuffix(\"```\").strip()\n",
"synthetic_data = json.loads(json_str)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d26cbba7",
"metadata": {},
"outputs": [],
"source": [
"with open(INTERIM_DIR /'synthetic_datasets/synthetic_data.json', 'w') as f:\n",
" json.dump(synthetic_data, f)"
]
},
{
"cell_type": "markdown",
"id": "fc52b327",
"metadata": {},
"source": [
"### Generate dataset without mbpp dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b16137cb",
"metadata": {},
"outputs": [],
"source": [
"llm_response = llm.invoke([SystemMessage(content=avap_docs), PROMPT_NO_MBPP])\n",
"print(llm_response.content)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "80d207fa",
"metadata": {},
"outputs": [],
"source": [
"json_str = llm_response.content.removeprefix(\"```json\").removesuffix(\"```\").strip()\n",
"synthetic_data = json.loads(json_str)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13e53200",
"metadata": {},
"outputs": [],
"source": [
"with open(INTERIM_DIR /'synthetic_datasets/synthetic_data_no_mbpp.json', 'w') as f:\n",
" json.dump(synthetic_data, f)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "assistance-engine",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

74
scripts/pipelines/flows/elasticsearch_ingestion.py Normal file
View File

@ -0,0 +1,74 @@
import hashlib
import logging
import os
from pathlib import Path
from elasticsearch import Elasticsearch
from langchain_elasticsearch import ElasticsearchStore
from langchain_ollama import OllamaEmbeddings
from scripts.pipelines.tasks.chunks import build_chunks_from_folder
logger = logging.getLogger(__name__)
ELASTICSEARCH_LOCAL_URL = os.getenv("ELASTICSEARCH_LOCAL_URL")
OLLAMA_LOCAL_URL = os.getenv("OLLAMA_LOCAL_URL")
ELASTICSEARCH_INDEX = os.getenv("ELASTICSEARCH_INDEX")
OLLAMA_URL = os.getenv("OLLAMA_URL")
OLLAMA_EMB_MODEL_NAME = os.getenv("OLLAMA_EMB_MODEL_NAME")
PROJ_ROOT = Path(__file__).resolve().parents[3]
def elasticsearch_ingestion(
docs_folder_path: str = "ingestion",
es_request_timeout: int = 120,
es_max_retries: int = 5,
es_retry_on_timeout: bool = True,
distance_strategy: str = "COSINE",
):
logger.info("Starting Elasticsearch ingestion pipeline...")
logger.info(f"Using docs folder path: {PROJ_ROOT / docs_folder_path}")
chunks = build_chunks_from_folder(folder_path=PROJ_ROOT / docs_folder_path)
logger.info("Connecting to Elasticsearch...")
try:
es = Elasticsearch(
ELASTICSEARCH_LOCAL_URL,
request_timeout=es_request_timeout,
max_retries=es_max_retries,
retry_on_timeout=es_retry_on_timeout,
)
except:
logger.exception("Failed to connect to Elasticsearch.")
raise
logger.info("Instantiating embeddings model...")
try:
embeddings = OllamaEmbeddings(
base_url=OLLAMA_LOCAL_URL, model=OLLAMA_EMB_MODEL_NAME
)
except:
logger.exception("Failed to instantiate embeddings model.")
raise
logger.info(f"Uploading documents to index {ELASTICSEARCH_INDEX}...")
ElasticsearchStore.from_documents(
chunks,
embeddings,
client=es,
index_name=ELASTICSEARCH_INDEX,
distance_strategy=distance_strategy,
)
logger.info(f"Finished uploading documents to index {ELASTICSEARCH_INDEX}.")
logger.info(f"Total documents uploaded: {len(chunks)}.")
if __name__ == "__main__":
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
try:
elasticsearch_ingestion()
except Exception as exc:
logger.exception(exc)
raise

46
scripts/pipelines/tasks/chunks.py Normal file
View File

@ -0,0 +1,46 @@
import re
import hashlib
import uuid
from pathlib import Path
from typing import Any
from langchain_core.documents import Document
def clean_text(text: str) -> str:
text = text.replace("\u00a0", " ")
text = re.sub(r"\s+", " ", text).strip()
return text
def build_chunks_from_folder(
folder_path: str,
) -> list[Document]:
folder = Path(folder_path)
if not folder.exists() or not folder.is_dir():
raise ValueError(f"Invalid folder path: {folder_path}")
all_chunks: list[Document] = []
for file_path in folder.glob("*.txt"):
doc_text = file_path.read_text(encoding="utf-8")
if not doc_text.strip():
continue
metadata: dict[str, Any] = {
"source": file_path.name,
}
doc_text = clean_text(doc_text)
chunk = Document(
id=hashlib.md5(file_path.name.encode()).hexdigest(),
page_content=doc_text,
metadata={**metadata,}
)
all_chunks.append(chunk)
return all_chunks

4
uv.lock
View File

@ -1,5 +1,5 @@
version = 1 version = 1
revision = 2 revision = 3
requires-python = ">=3.11" requires-python = ">=3.11"
resolution-markers = [ resolution-markers = [
"python_full_version >= '3.14' and sys_platform == 'win32'", "python_full_version >= '3.14' and sys_platform == 'win32'",
@ -290,6 +290,7 @@ dependencies = [
dev = [ dev = [
{ name = "accelerate" }, { name = "accelerate" },
{ name = "beir" }, { name = "beir" },
{ name = "boto3" },
{ name = "evidently" }, { name = "evidently" },
{ name = "jupyter" }, { name = "jupyter" },
{ name = "langfuse" }, { name = "langfuse" },
@ -328,6 +329,7 @@ requires-dist = [
dev = [ dev = [
{ name = "accelerate", specifier = ">=1.12.0" }, { name = "accelerate", specifier = ">=1.12.0" },
{ name = "beir", specifier = ">=2.2.0" }, { name = "beir", specifier = ">=2.2.0" },
{ name = "boto3", specifier = ">=1.42.58" },
{ name = "evidently", specifier = ">=0.7.20" }, { name = "evidently", specifier = ">=0.7.20" },
{ name = "jupyter", specifier = ">=1.1.1" }, { name = "jupyter", specifier = ">=1.1.1" },
{ name = "langfuse", specifier = "<=3" }, { name = "langfuse", specifier = "<=3" },