Cognifloyd's Cognifire: Scattered design thoughts

So, I have notes scattered everywhere and I'm somewhat lost. I'm using this post to try and collect my notes, and get a better idea of where I am and where I'm going. Though you're welcome to browse through this, I don't expect these notes to make sense to anyone other than me.

BuilderFoundation provides a MetaModel for packages that other Builders will extend.

A Package can be built like FormDefinitions from TYPO3.Form

FormDefinitions have:

finishers <array>
processingRules <array>
elementsByIdentifier <array>
eleementsDefaultValue <array>
formFieldTypeManager <injected class>
validatorPresets <array>
finisherPresets <array>
setOptions (stuff that works with info from a YAML)

setRendererClassName
setRenderingOption
createFinisher

The FormFactory builds FormDefinitions
Use the name "presets" which would work as an options layer in general, or a "themes" layer in TemplateBuilder (a preset is a name + configuration; or iow it is a named set of configuration)
Based on YAML files
Using a yamlPersistenceManager

YAML files

Use superTypes inheritance concept (like in Forms and TYPO3CR: NodeTypes.yaml) to allow for things like mixins
Always get converted to arrays
considered using TypoScript, but YAML seems to fit how I'll use this better. (YAML is for data serialization. TypoScript helps in processing things)
YAML processors should always have at least one of dump() or load()
Each package has a configuration file in YAML
boilerplates and settings repositories should be added from the yaml (Cognifire\BuilderFoundation\boilerplateRepositories & derivativeRepositories)
Current configuration files

Settings.yaml
Routes.yaml
Objects.yaml
Policy.yaml
PackageStates.php
Caches.yaml
Views.yaml

It would be nice to be able to breakup boilerplate configurations so that there's one preset per file (or somethin like that). <whatever base name>.<preset name>.yaml
Will need a schema:

AbstractPackage: Boilerplates and Derivatives

not packageKey identified, but w/ UUID because packageKey might change several times during package build process
What gets stored in the package object is metadata about it's creation

a declarative set of steps--if you will--to allow the builder to recreate the same derivative but with another package key
Only part of this can be found out by parsing directory structures.
Instead, you

list Packages/<target repo>/*
read each <package>/Configuration/<yaml file>

Yaml file names (not satisfied with these names)

~~Themes.yaml~~ (Builders might want to provide their own yaml file, like the templateBuilder where the word "themes" makes a lot of sense)
Possible names in Boilerplates:

~~Builder.Boilerplate.yaml~~
Boilerplate.yaml

Declared components (all available)

~~Boilerplate.presets.yaml~~
~~Boilerplate.presets.<something>.yaml~~

Possible names in Derivatives:

~~Builder.Derivative.yaml~~
Derivative.yaml
~~Boilerplate.Integrated.yaml~~

Declare which components were used and from where without introducing a dependency on the boilerplate

~~Builder.Template.yaml or Builder.Package.yaml~~

contains the presets or the final configuration, how all of declared components were put together
Builder specific?

Boilerplate.yaml and Derivative.yaml can be a collection of YAML files to keep the component definitions in the same folder as the component. Everything in that Boilerplate would apply only to that path. With one exception: Resources/Private/[Boilerplate || Derivative].yaml applies to the whole package. They should be restricted to the Resources folder, though, so that we don't have YAML files in the Classes/ or Tests/ folders. Boilerplate.yaml and Derivative.yaml should also not be in Settings/ as that is reserved for Presets.*.yaml for builder stuff as well as other configuration that is not builder-related.
Basically, this means that Boilerplates will store the snippets they use to build files in the Resources folder.

Boilerplate vs Derivative

Package Repository (Boilerplate vs Application)
Derivative

Root files (composer.json...)
Resources
Configuration
Documentation
Classes
Tests
Scripts
Migrations
Meta

Yaml files, rethought

<package>/Resources/Private/Boilerplate.yaml
Declares which components/resources are available in boilerplate including:

Which builder is required to work with a particular set of files. Or maybe they would declare filetypes like x.html is fluid and y.html is handlebars so that the appropriate Builder can be used, and the Builder would say which filetypes it works with)
A versioning "namespace" of sorts, so that a boilerplate can include multiple versions of layouts/templates/partials (thinking of TwitterBootstrap v2 and v3). This would assist the migration service
Perhaps a list of presets that are available in the Boilerplate (though that should probably be based on what is in the Settings/)

<package>/Resources/Private/Derivative.yaml

Declares which components & presets have been used

from which boilerplate
and which boilerplate version (to assist with migrations)

Perhaps a list of which manual changes have been detected
This file would have the Eel selectors to say "insert [component] at [location] in [file]" including components from boilerplates and components that are added manually. This should be verbose enough that a Builder could regenerate the templates from the boilerplates (for the custom components [the 'changeset in Boilerplate math], maybe the builder could pull them out and store them in another Resources/Private/ folder so that it can keep track of manual changes.)
Though this should be human readable, only Builders will edit it. I can imagine people tweaking an Eel statement. But most of the file is for Builder metadata persistence.

<package>/Settings/
Presets.[BuilderSpecific].[PresetName].yaml
A preset configures a set of components to work together. This is the Builders' "options" layer. It does not define how the files are combined. That belongs in the Boilerplate.yaml (for what can be combined) or Deriative.yaml (for what has been combined) files.
Should the BuilderSpecific part be singular or plural? You can have multiple presets in a Presets.Builder.yaml file, or break them out into individual Presets.Builder.Preset1.yaml and Presets.Builder.Preset2.yaml files.
Someone could edit a Preset manually and use the Builder to make the change in their templates. These should be human-friendly. They would have to fire up the Builder to make the builder adjust whatever files are built with that preset (like changing a color in a LESS file).
Examples of possible uses per builder (just examples, most are not part of GSoC project):

PackageBuilder

Presets.Package.CMS.yaml
Presets.Package.Flow.yaml
Presets.Package.Symfony.yaml (example extension of PackageBuilder)

TemplateBuilder

Presets.Theme.yaml
Presets.Theme.FixedLayout.yaml
Presets.Theme.AwesomeGreen.yaml
Presets.Theme.MyThemeName.yaml
Presets.Fluid.yaml (for stuff that doesn't quite fit the term "Theme"

TypoScriptBuilder

Presets.TypoScript.yaml

EmberBuilder or JavaScriptBuilder (Works with JavaScript stuff like Ember and Handlebars)

Presets.Ember.yaml
Presets.Ember.Handlebars.yaml
Presets.Handlebars.yaml
Presets.JS.yaml (maybe everything could be Presets.JS.*.yaml like Presets.JS.Ember.yaml and Presets.JS.Handlebars.yaml)

FormBuilder (if it switches to using BuilderFoundation at its core)

Presets.Form.yaml
Presets.Form.Cli.yaml
Presets.Form.Bootstrap.yaml

TranslationBuilder (works with .xlf files)

Presets.Translation.yaml (not sure how a TranslationBuilder would need Presets, but it's possible)

SchemaBuilder (a Resource that needs to be built. Different Builders might want to create a schema for what is allowed in their Presets.[Builder].*.yaml files, and they might not want to craft the schemas by hand)
OntologyBuilder/TaxonomyBuilder (Not sure this one will work with packages or boilerplates, but, the Builder concept still applies)

~~Yaml file contents Boilerplate vs Derivative (extendable yaml schema)~~

~~Boilerplate~~

~~relevant builder (like template builder)~~

~~components that can be built~~

~~include[superType]~~
~~Variations of each component~~
~~Options that apply to each component~~

~~presets~~

~~components 1,2,5~~

~~variations such and so~~
~~options~~

~~TemplateBuilder: (example specific builder)~~

~~Declare what is avilable~~

~~Components:~~

~~Variations~~
~~Options~~

~~Variations~~
~~Options (Theme in TemplateBuilder)~~

~~color1:~~
~~color2:~~
~~width:~~

~~Presets are pre-configured defaults / collections of components and options~~

~~Presets / <presetname>~~

~~components: 1, 2, 3~~
~~variation: A, B~~
~~options~~

~~Derivative~~

~~Has one preset that lists which~~

~~boilerplate components are in use~~
~~how they are configured~~
~~what options are set~~

~~Boilerplate.Integrated.yaml~~

~~declares components used from each boierplate~~

~~copied from boilerplate w/ boilerplate version~~

Spl*Iterators [Directory, FileSystem, Glob]:

Relationships

GlobIterator extends FileSystemIterator extends DirectoryIterator
RecursiveDirectoryIterator extends FileSystemIterator as well
GlobIterator is one level w/ glob pattern search
DirectoryIterator is one level
FileSystemIterator is just like DirectoryIterator but with flags that give more control over what current and key return.
RecursiveDirectoryIterator does many levels

Interesting FileSystemIterator Flags

::CURRENT_AS_PATHNAME
::KEY_AS_PATHNAME
::KEY_AS_FILENAME
::FOLLOW_SYMLINKS
::NEW_CURRENT_AND_KEY (KEY_AS_FILENAME & CURRENT_AS_FILEINFO)
::SKIP_DOTS
::UNIX_PATHS (I'll want this one)
Current could return Pathname, or Filename or Fileinfo (which is a SplFileInfo object)

Key is just a numeric array key

SplFileInfo interesting methods (also available on DirectoryIterator and FileSystemIterator)

getBasename (can omit a suffix from path & filename)
getFilename
getPath (w/o filename)
openFile (creates SplFileObject to iterate ovver a file w/ fwrite, fpassthru...)
getRealPath (resolve links)
getExtension
getType (file, link, dir)

Streams

file:// -- for file access
glob://*.png -- for listing dir contents that match the glob
data: -- to encode stuff straight in files and treat it as a file/stream (RFC 2397)

data:text/plan,$data
data:text/plain;base64,$data
data:image/jpeg;base64;$image

resource:// -- flow specific only for stuff in resources of a particular package

The purpose of this is not to hide that something is a file, but to hide where it is stored (so that Flow can do the linking magic with all the public stuff in the Web/_Resources folder

Persistence

Flow injects the persistenceManagerInterface
There's no way to say which one flow will use, instead it is specified in objects.yaml (doctrine by default)
Mutliple concurrent Persistence backends won't be available until Flow 2.2 (or Flow 3 depending on how backwards compatible it is

define on repository basis what kind of DB to persist it in
This is not implemented yet due to lack of funding
I would rather reuse the flow persistence stuff where it makes sense, but I have to decouple everything until this is implemented. I'll copy lots of stuff to make it work. The thing is, a DB doesn't make any sense in my packages, even though I'm persisting stuff, the code assumes that persistence is always to a DB.
workaround is to put persistence stuff in the repository and not use @Flow\Entity and the like

"Models do not know (a lot) about Persistence (internals)"
Repository responsible for:

Saving object
persisting changes to object throughout lifetime (update DB as needed)
provide QL

handle DB retrieval
ensures only one instance in memory

provides the aggregate root

Comparing Radmiraal.CouchDB (doctrine branch) with normal doctrine persistence

@ODM\Document instead of @ORM\Entity or @Flow\Entity
replaces persistence manager w/ document manager
queries

no query on arbitrary properties
drops queries in favor of lucenQueries
setQueryMatchValue

both use persistenceManager

->getObjectByIdentifier in generic
->getIdentifierByObject in radmiraal

persistenceManager vs documentManager method names

add; persist
update; merge
remove; remove
createQueryForType; [createLuceneQuery, createNativeQuery, createQuery]

Doctrine's PersistenceManager uses the Annotations Driver

Annotations

Entity

map the repository / set CustomRepositoryClass
markReadOnly

loadMetaDataForClass

evaluatePropertyAnnotations

maps to column names, OneToOne, OneToMany, ManyToOne, ManyToMany

evaluateOverridesAnnotations
evaluateLifeCycleAnnotations

public methods

PrePersist
PostPersist
PreUpdate
PostUpdate
PreRemove
PostRemove
PostLoad
PreFlush

isTransient (yes if not entity or value object)

I want to use Flow/Validate
PersistenceManager fires

Yaml Configuration

All combined/cached like views.yaml

How other things interact w/ files

Parts of flow that work with files

Reflection uses cache to interact w/ Files
Fluid uses cache in compiler, and strings get passed to parser (it doesn't work w/ files)
TypoScript also parses strings, not files
Kickstarter