Monday, June 10, 2013

Scattered design thoughts

So, I have notes scattered everywhere and I'm somewhat lost. I'm using this post to try and collect my notes, and get a better idea of where I am and where I'm going. Though you're welcome to browse through this, I don't expect these notes to make sense to anyone other than me.



BuilderFoundation provides a MetaModel for packages that other Builders will extend.

A Package can be built like FormDefinitions from TYPO3.Form
  • FormDefinitions have:
    • finishers <array>
    • processingRules <array>
    • elementsByIdentifier <array>
    • eleementsDefaultValue <array>
    • formFieldTypeManager <injected class>
    • validatorPresets <array>
    • finisherPresets <array>
    • setOptions (stuff that works with info from a YAML)
      • setRendererClassName
      • setRenderingOption
      • createFinisher
  • The FormFactory builds FormDefinitions
  • Use the name "presets" which would work as an options layer in general, or a "themes" layer in TemplateBuilder (a preset is a name + configuration; or iow it is a named set of configuration)
  • Based on YAML files
  • Using a yamlPersistenceManager
YAML files
  • Use superTypes inheritance concept (like in Forms and TYPO3CR: NodeTypes.yaml) to allow for things like mixins
  • Always get converted to arrays
  • considered using TypoScript, but YAML seems to fit how I'll use this better. (YAML is for data serialization. TypoScript helps in processing things)
  • YAML processors should always have at least one of dump() or load()
  • Each package has a configuration file in YAML
  • boilerplates and settings repositories should be added from the yaml (Cognifire\BuilderFoundation\boilerplateRepositories & derivativeRepositories)
  • Current configuration files
    • Settings.yaml
    • Routes.yaml
    • Objects.yaml
    • Policy.yaml
    • PackageStates.php
    • Caches.yaml
    • Views.yaml
  • It would be nice to be able to breakup boilerplate configurations so that there's one preset per file (or somethin like that). <whatever base name>.<preset name>.yaml
  • Will need a schema:
AbstractPackage: Boilerplates and Derivatives
  • not packageKey identified, but w/ UUID because packageKey might change several times during package build process
  • What gets stored in the package object is metadata about it's creation
    • a declarative set of steps--if you will--to allow the builder to recreate the same derivative but with another package key
    • Only part of this can be found out by parsing directory structures.
    • Instead, you
      • list Packages/<target repo>/*
      • read each <package>/Configuration/<yaml file>
  • Yaml file names (not satisfied with these names)
    • Themes.yaml (Builders might want to provide their own yaml file, like the templateBuilder where the word "themes" makes a lot of sense)
    • Possible names in Boilerplates:
      • Builder.Boilerplate.yaml
      • Boilerplate.yaml
        • Declared components (all available)
      • Boilerplate.presets.yaml
      • Boilerplate.presets.<something>.yaml
    • Possible names in Derivatives:
      • Builder.Derivative.yaml
      • Derivative.yaml
      • Boilerplate.Integrated.yaml
        • Declare which components were used and from where without introducing a dependency on the boilerplate
      • Builder.Template.yaml or Builder.Package.yaml
        • contains the presets or the final configuration, how all of declared components were put together
        • Builder specific?
    • Boilerplate.yaml and Derivative.yaml can be a collection of YAML files to keep the component definitions in the same folder as the component. Everything in that Boilerplate would apply only to that path. With one exception: Resources/Private/[Boilerplate || Derivative].yaml applies to the whole package. They should be restricted to the Resources folder, though, so that we don't have YAML files in the Classes/ or Tests/ folders. Boilerplate.yaml and Derivative.yaml should also not be in Settings/ as that is reserved for Presets.*.yaml for builder stuff as well as other configuration that is not builder-related.
    • Basically, this means that Boilerplates will store the snippets they use to build files in the Resources folder.
  • Boilerplate vs Derivative
    • Package Repository (Boilerplate vs Application)
    • Derivative
      • Root files (composer.json...)
      • Resources
      • Configuration
      • Documentation
      • Classes
      • Tests
      • Scripts
      • Migrations
      • Meta
  • Yaml files, rethought
    • <package>/Resources/Private/Boilerplate.yaml
      Declares which components/resources are available in boilerplate including:
      • Which builder is required to work with a particular set of files. Or maybe they would declare filetypes like x.html is fluid and y.html is handlebars so that the appropriate Builder can be used, and the Builder would say which filetypes it works with)
      • A versioning "namespace" of sorts, so that a boilerplate can include multiple versions of layouts/templates/partials (thinking of TwitterBootstrap v2 and v3). This would assist the migration service
      • Perhaps a list of presets that are available in the Boilerplate (though that should probably be based on what is in the Settings/)
    • <package>/Resources/Private/Derivative.yaml
      • Declares which components & presets have been used
        • from which boilerplate
        • and which boilerplate version (to assist with migrations)
      • Perhaps a list of which manual changes have been detected
      • This file would have the Eel selectors to say "insert [component] at [location] in [file]" including components from boilerplates and components that are added manually. This should be verbose enough that a Builder could regenerate the templates from the boilerplates (for the custom components [the 'changeset in Boilerplate math], maybe the builder could pull them out and store them in another Resources/Private/ folder so that it can keep track of manual changes.)
      • Though this should be human readable, only Builders will edit it. I can imagine people tweaking an Eel statement. But most of the file is for Builder metadata persistence.
    • <package>/Settings/
      Presets.[BuilderSpecific].[PresetName].yaml

      A preset configures a set of components to work together. This is the Builders' "options" layer. It does not define how the files are combined. That belongs in the Boilerplate.yaml (for what can be combined) or Deriative.yaml (for what has been combined) files.
      Should the BuilderSpecific part be singular or plural? You can have multiple presets in a Presets.Builder.yaml file, or break them out into individual Presets.Builder.Preset1.yaml and Presets.Builder.Preset2.yaml files.
      Someone could edit a Preset manually and use the Builder to make the change in their templates. These should be human-friendly. They would have to fire up the Builder to make the builder adjust whatever files are built with that preset (like changing a color in a LESS file).
      Examples of possible uses per builder (just examples, most are not part of GSoC project):
      • PackageBuilder
        • Presets.Package.CMS.yaml
        • Presets.Package.Flow.yaml
        • Presets.Package.Symfony.yaml (example extension of PackageBuilder)
      • TemplateBuilder
        • Presets.Theme.yaml
        • Presets.Theme.FixedLayout.yaml
        • Presets.Theme.AwesomeGreen.yaml
        • Presets.Theme.MyThemeName.yaml
        • Presets.Fluid.yaml (for stuff that doesn't quite fit the term "Theme"
      • TypoScriptBuilder
        • Presets.TypoScript.yaml
      • EmberBuilder or JavaScriptBuilder (Works with JavaScript stuff like Ember and Handlebars)
        • Presets.Ember.yaml
        • Presets.Ember.Handlebars.yaml
        • Presets.Handlebars.yaml
        • Presets.JS.yaml (maybe everything could be Presets.JS.*.yaml like Presets.JS.Ember.yaml and Presets.JS.Handlebars.yaml)
      • FormBuilder (if it switches to using BuilderFoundation at its core)
        • Presets.Form.yaml
        • Presets.Form.Cli.yaml
        • Presets.Form.Bootstrap.yaml
      • TranslationBuilder (works with .xlf files)
        • Presets.Translation.yaml (not sure how a TranslationBuilder would need Presets, but it's possible)
      • SchemaBuilder (a Resource that needs to be built. Different Builders might want to create a schema for what is allowed in their Presets.[Builder].*.yaml files, and they might not want to craft the schemas by hand)
      • OntologyBuilder/TaxonomyBuilder (Not sure this one will work with packages or boilerplates, but, the Builder concept still applies)
  • Yaml file contents Boilerplate vs Derivative (extendable yaml schema)
    • Boilerplate
      • relevant builder (like template builder)
        • components that can be built
          • include[superType]
          • Variations of each component
          • Options that apply to each component
      • presets
        • components 1,2,5
          • variations such and so
          • options
      • TemplateBuilder: (example specific builder)
        • Declare what is avilable
          • Components:
            • Variations
            • Options
          • Variations
          • Options (Theme in TemplateBuilder)
            • color1:
            • color2:
            • width:
        • Presets are pre-configured defaults / collections of components and options
          • Presets / <presetname>
            • components: 1, 2, 3
            • variation: A, B
            • options
    • Derivative
      • Has one preset that lists which
        • boilerplate components are in use
        • how they are configured
        • what options are set
      • Boilerplate.Integrated.yaml
        • declares components used from each boierplate
          • copied from boilerplate w/ boilerplate version
Spl*Iterators [Directory, FileSystem, Glob]:
  • Relationships
    • GlobIterator extends FileSystemIterator extends DirectoryIterator
    • RecursiveDirectoryIterator extends FileSystemIterator as well
    • GlobIterator is one level w/ glob pattern search
    • DirectoryIterator is one level
    • FileSystemIterator is just like DirectoryIterator but with flags that give more control over what current and key return.
    • RecursiveDirectoryIterator does many levels
  • Interesting FileSystemIterator Flags
    • ::CURRENT_AS_PATHNAME
    • ::KEY_AS_PATHNAME
    • ::KEY_AS_FILENAME
    • ::FOLLOW_SYMLINKS
    • ::NEW_CURRENT_AND_KEY (KEY_AS_FILENAME & CURRENT_AS_FILEINFO)
    • ::SKIP_DOTS
    • ::UNIX_PATHS (I'll want this one)
    • Current could return Pathname, or Filename or Fileinfo (which is a SplFileInfo object)
    • Key is just a numeric array key
  • SplFileInfo interesting methods (also available on DirectoryIterator and FileSystemIterator)
    • getBasename (can omit a suffix from path & filename)
    • getFilename
    • getPath (w/o filename)
    • openFile (creates SplFileObject to iterate ovver a file w/ fwrite, fpassthru...)
    • getRealPath (resolve links)
    • getExtension
    • getType (file, link, dir)
  • Streams
    • file:// -- for file access
    • glob://*.png -- for listing dir contents that match the glob
    • data: -- to encode stuff straight in files and treat it as a file/stream (RFC 2397)
      • data:text/plan,$data
      • data:text/plain;base64,$data
      • data:image/jpeg;base64;$image
    • resource:// -- flow specific only for stuff in resources of a particular package
      • The purpose of this is not to hide that something is a file, but to hide where it is stored (so that Flow can do the linking magic with all the public stuff in the Web/_Resources folder
Persistence
  • Flow injects the persistenceManagerInterface
  • There's no way to say which one flow will use, instead it is specified in objects.yaml (doctrine by default)
  • Mutliple concurrent Persistence backends won't be available until Flow 2.2 (or Flow 3 depending on how backwards compatible it is
    • define on repository basis what kind of DB to persist it in
    • This is not implemented yet due to lack of funding
    • I would rather reuse the flow persistence stuff where it makes sense, but I have to decouple everything until this is implemented. I'll copy lots of stuff to make it work. The thing is, a DB doesn't make any sense in my packages, even though I'm persisting stuff, the code assumes that persistence is always to a DB.
    • workaround is to put persistence stuff in the repository and not use @Flow\Entity and the like
  • "Models do not know (a lot) about Persistence (internals)"
  • Repository responsible for:
    • Saving object
    • persisting changes to object throughout lifetime (update DB as needed)
    • provide QL
      • handle DB retrieval
      • ensures only one instance in memory
    • provides the aggregate root
  • Comparing Radmiraal.CouchDB (doctrine branch) with normal doctrine persistence
    • @ODM\Document instead of @ORM\Entity or @Flow\Entity
    • replaces persistence manager w/ document manager
    • queries
      • no query on arbitrary properties
      • drops queries in favor of lucenQueries
      • setQueryMatchValue
    • both use persistenceManager
      • ->getObjectByIdentifier in generic
      • ->getIdentifierByObject in radmiraal
    • persistenceManager vs documentManager method names
      • add; persist
      • update; merge
      • remove; remove
      • createQueryForType; [createLuceneQuery, createNativeQuery, createQuery]
  • Doctrine's PersistenceManager uses the Annotations Driver
    • Annotations
      • Entity
        • map the repository / set CustomRepositoryClass
        • markReadOnly
    • loadMetaDataForClass
      • evaluatePropertyAnnotations
        • maps to column names, OneToOne, OneToMany, ManyToOne, ManyToMany
      • evaluateOverridesAnnotations
      • evaluateLifeCycleAnnotations
        • public methods
          • PrePersist
          • PostPersist
          • PreUpdate
          • PostUpdate
          • PreRemove
          • PostRemove
          • PostLoad
          • PreFlush
      • isTransient (yes if not entity or value object)
    • I want to use Flow/Validate
    • PersistenceManager fires
      • Yaml Configuration
        • All combined/cached like views.yaml

How other things interact w/ files
  • Parts of flow that work with files
    • Reflection uses cache to interact w/ Files
    • Fluid uses cache in compiler, and strings get passed to parser (it doesn't work w/ files)
    • TypoScript also parses strings, not files
    • Kickstarter
      • uses $templatePathAndFilename = resource://TYPO3.Kickstart/Private/...
      • shoves into renderTemplate()
      • then generateFile()
        • File::createDirectoryRecursviely
      • uses reflection for php files
      • hard codes important paths
        • Classes
        • Tests
        • Resources
      • Validate PackageKey ; PackageManager->isPackageAvailable
  • TYPO3\Flow\Utility
    • Environment
      • $this->environment->getPathToTemporaryDirectory()
    • Files
      • ::getUnixStylePath($cacheDirectory)
      • ::concatenatePaths($cacheDirectory)
      • ::readDirectoryRecursively($cacheDirectory)
      • ::createDirectoryRecursively($cacheDirectory)
      • ::emptyDirectoryRecursively($cacheDirectory)
  • php funcs
    • file_exists($pathAndFillename)
    • unlink($pathAndFilename)
    • file_get_contents($pathAndFilename)
    • file_put_contents($tempCacheEntryPathAndFilename, $data)
    • rename($tempCacheEntryPathAndFilename, $newPathAndFilename)
    • filesize($cacheEntryPathAndFilename)
    • filemtime($file)
    • new \DirectoryIterator ($this->cacheDirectory)
      • ->valid()
      • ->next()
      • ->isDot()
    • glob($pattern) find pathname matching the pattern
    • $fileHandle=fopen($this->logFileUrl, 'ab')
    • dirname($logFileUrl)
    • is_dir($)
    • is_link($)
    • stream_get_meta_data($fileHandle)
    • fclose($)
    • chmod( , )
    • touch()
    • fputs()