܍~c$dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZddlmZmZmZmZmZmZmZddlZdd lmZmZdd lmZdd l m!Z!m"Z"m#Z# dd l$m%Z&e&Z%n #e'$re(Z%YnwxYwee!d Z!ej)e*Z+dZ,dZ-ddddZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6dZ7erddlZddl8m9Z9m:Z:Gdd e:Z;neee>Z?dd%d&d'e?d(ee@d)e@d*e@fd+ZAGd,d-eBZCd.ZDdfd/ee@d*e@fd0ZEe=fdeFd1eZLGd?d@ZMdddAdBez)_format_schema_problems..psQXXZZr/)joinmap)rAr?r@ formatteds r0_format_schema_problemsrJjsA s#7#7IIJJI +*y** r/cVeZdZdZ ddeedeeffd ZdefdZxZ S) SchemaValidationErrorz.Az)B)B%Br/c$t|SrDr{r|s r0rFz_anyOf..Z:66r/, is not valid under any of the given schemascontextN)r_r enumeratelistdescendfilterextend) validatoranyOfrf_schemarpr all_errorsall_deprecationsindex subschemaall_errserrs deprecationss ` r0_anyOfrs<+*****J%e,,  y   h u  E E  FBBBBHMMNN 6666 A A     # #L 1 1 1 E$o@H J       r/c#bKddlm}t|}g}g}|D]\} } t| | } tt fd| } tt fd| } | s| }|| n(|| |d|Vfd|D}|rH||d d |D}|d |Vd S|Ed {Vd S) zJsonschema validator for `oneOf`. It treats occurrences of `error_type` as non-errors, but yield them for external processing. Useful to process schema annotations, as `deprecated`. rrrxc&t| SrDr{r|s r0rFz_oneOf..r~r/c$t|SrDr{r|s r0rFz_oneOf..rr/rrcFg|]\}}||Sr.)is_valid).0isrfrs r0 z_oneOf..s2NNN1i.@.@1.M.MN!NNNr/r=c34K|]}t|VdSrD)repr)rros r0 z_oneOf.. s(@@6$v,,@@@@@@r/z is valid under each of N) r_rrrrrrappendrG)roneOfrfrrpr subschemasrrrrrrr first_valid more_validreprss` ` ` r0_oneOfrs+*****5!!JJ&  y   h u  E E  FBBBBHMMNN 6666 A A   #K  # #L 1 1 1 E$o@H J      ONNNN NNNJ$+&&& @@Z@@@@@o.6hh >       $#########r/cddlm}m}ddlm}t |j}d|d<ddi|dd <i}t|d r%|j dt}d |i}n|j }ttf|d<d |i}t|j}t |t"<t$|d <t&|d<|d||dd|}dd} | |_||fS)zGet metaschema validator and format checker Older versions of jsonschema require some compatibility changes. @returns: Tuple: (jsonschema.Validator, FormatChecker) @raises: ImportError when jsonschema is not present r)r] FormatChecker)createFadditionalPropertiestyper^ propertieslabelra type_checker default_typesrrdraft4) meta_schema validatorsversionNcptd|||}t|dduS)zgOverride version of `is_valid`. It does ignore instances of `SchemaDeprecationError`. c.t|t SrD)rcr2r}s r0rFz.is_valid..Ps*)r/N)r iter_errorsnext)r;rfr__errorss r0rz*get_jsonschema_validator..is_validJsI      Xw / /    FD!!T))r/r.rD)r_r]rjsonschema.validatorsrr META_SCHEMAhasattrraredefinerg DEFAULT_TYPESr+rddict VALIDATORSrvDEPRECATED_KEYrrr) r]rrstrict_metaschemavalidator_kwargsrtypesrcloudinitValidatorrs r0get_jsonschema_validatorrsY:9999999,,,,,, !!<==05,-17/Al#G,// 4&3<< +   L  -,h+U3o011J!6J~ Jw Jw%   * * * *#+  ..r/c ,ddlm} ||dS#|$ro}|rGtt dd|jD|jg|t d|Yd}~dSd}~wwxYw)a Validate provided schema meets the metaschema definition. Return strict Validator and FormatChecker for use in validation @param validator: Draft4Validator instance used to validate the schema @param schema: schema to validate @param throw: Sometimes the validator and checker are required, even if the schema is invalid. Toggle for whether to raise SchemaValidationError or log warnings. @raises: ImportError when jsonschema is not present @raises: SchemaValidationError when the schema is invalid r) SchemaErrorrjc,g|]}t|Sr.r+rrEs r0rz3validate_cloudconfig_metaschema..ts!;!;!;Q#a&&!;!;!;r/rMzGMeta-schema validation failed, attempting to validate config anyway: %sN) jsonschema.exceptionsr check_schemarLr4rGr5r6LOGwarning)rrothrowrerrs r0validate_cloudconfig_metaschemar\s211111 v&&&&&      '!!;!;#(!;!;!;<.sQVr/keyrjc,g|]}t|Sr.rrs r0rz/validate_cloudconfig_schema..s444AQ444r/z"Deprecated cloud-config provided:  r>zInvalid cloud-config provided: zeInvalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.) get_schemarrr`rdebugsortedrrGr5r4r6rcr2rJrrL)rrorrrrrrrrrrr5problemr6detailss r0validate_cloudconfig_schemars8~.F.H.H+ ]   +"F%      IJJJ#"6--//JJJIF#%L --f55;K;KLLLxx4444455 u}557 e3 4 4  G #LL g FFL) 8    G :6:\:#FL999   -9GG)  G  s%8$A A c eZdZdededefdZededeedefdZ d e fd Z e dd eed eedeede dede f dZ deedededeefdZde de deeeffdZdS) _Annotator cloudconfigoriginal_content schemamarksc0||_||_||_dSrD) _cloudconfig_original_content _schemamarks)r;rrrs r0rTz_Annotator.__init__s" (!1'r/r"contentr7c>d|}d|d|dS)Nr# z: ------------- rQ)rG)r"rbodys r0 _build_footerz_Annotator._build_footers,yy!!6E66D6666r/rAcptt}|D]\}}tjd|}|r@|\}}|t ||n(d}||j|||d|||}|S)Nz&format-l(?P\d+)\.c(?P\d+).*zLine {line} column {col}: {msg})linecolru) rrrematchgroupsintrrr<)r;rAerrors_by_liner5rurrrs r0_build_errors_by_linez _Annotator._build_errors_by_lines$T***  KT3HFMME D!LLNN cs4yy)005555t067>>sCCC7>>3C?r/rPproblemslabelsfooterr label_prefixc|D]=}||}|||d|d||dz }>|S)Nrr9)r)rrrrrrrs r0 _add_problemsz_Annotator._add_problemssh   G#,U,,E MM% MM1u1111 2 2 2 QJEE r/linesrdeprecations_by_linec g}g}g}d}d}t|dD]\} } || } || } | s| reg} | | ||d}| | ||d}|| dzd| z~|| |t fdt dd |fd |ff|S) NrE)rDz # ,cj|SrD)r)seqr;s r0rFz._Annotator._annotate_content..s.D.4r/c,t|dS)Nr)rW)r s r0rFz._Annotator._annotate_content..sSV r/Errors Deprecations)rrrrGrrHr)r;rrrannotated_content error_footerdeprecation_footer error_indexdeprecation_index line_numberrrrrs` r0_annotate_contentz_Annotator._annotate_contents_ "$ (* !*5!!4!4 / / K#K0F/ #7  yy*+++r/N)rP)r(r)r*rrdrT staticmethodr+r rrZrrrrr rr.r/r0rrs(( ( ((((7S7497777\7^    s) S  S        \ *!Cy*!*!# *! c *!*!*!*!X,%,,, sEz  ,,,,,,r/rrMrNrrrrMrNcTt||||pg|pgS)aReturn contents of the cloud-config file annotated with schema errors. @param cloudconfig: YAML-loaded dict from the original_content or empty dict if unparseable. @param original_content: The contents of a cloud-config file @param schemamarks: Dict with schema marks. @param schema_errors: Instance of `SchemaProblems`. @param schema_deprecations: Instance of `SchemaProblems`. @return Annotated schema )rr)rrrrMrNs r0annotated_cloudconfig_filerAs9& k#3[ A A J J06B  r/c |tjdkrtdtg}|d|t |jdd }nRtj |s"td |t |d }| tsttd d |tg}t!|}|r%t#t%i|i|j | |rt)j|\}}nt)j|}i}n#t.j$r} dx} } d} t3| dr!t5| drt5| d} n0t3| dr t5| drt5| d} | r| jdz} | jdz} td | | d |t;| g}t!|}|r%t#t%i|i|j || d} ~ wwxYwt=|t>s|std tA||dddS#t $rt} |r,t#t%|||| j| j!n&tE| j!dd} t#| | #rYd} ~ dSd} ~ wwxYw)aValidate cloudconfig file adheres to a specific jsonschema. @param config_path: Path to the yaml cloud-config file to parse, or None to default to system userdata from Paths object. @param schema: Dict describing a valid jsonschema to validate against. @param annotate: Boolean set True to print original config file with error annotations on the offending lines. @raises SchemaValidationError containing any of schema_errors encountered. @raises RuntimeError when config_path does not exist. Nrz?Unable to read system userdata as non-root user. Try using sudo)ds_depstrust)existing cloud_configF)rzConfigfile {0} does not existz format-l1.c1z"File {0} needs to begin with "{1}"rr context_mark problem_markzformat-l{line}.c{col})rrzFile {0} is not valid yaml. {1}z Cloud-config is not a YAML dict.T)rrrrRr=r>)$osgetuid RuntimeErrorrfetch consume_datarpaths get_ipathr5existsr< startswithCLOUD_CONFIG_HEADERr4rrLprintrrMrload_with_marksloadyaml YAMLErrorrgetattrrcolumnr+rcrrrNrJrX) config_pathrorinitrrrrmarksr}rr5markr6s r0validate_cloudconfig_filer:Ys 9;;!  " B G $$$ DJ00@@OOOw~~k** /66{CC K666   1 2 2 4;;!4!;!;!=!=   &f--   *53F       !)!9'!B!B K"-00KE Nv 1n % % .'!^*D*D .1n--DD Q ' ' .GA~,F,F .1n--D  %9q=D[1_F '..Df.EE188c!ffMM    &f--   *53F    /0 k4 ( (C CABB B# u       !   *"#/()(=     .%;G 'NNN <<>>        %s2.0F J20C=J--J2K11 M/;A)M**M/c&|dkrdS|dkrdSdS)zProvide a sorting weight for documentation of property types. Weight values ensure 'array' sorted after 'object' which is sorted after anything else which remains unsorted. arrayobjectrrr.)values r0_sort_property_orderr@s)  q (  q 1r/c#K|D]M}t|tr2t|ttfst |Ed{VI|VNdSrD)rcrr+rd_flatten)xsxs r0rBrBsr  a " " :a#u+F+F {{ " " " " " " " "GGGG r/ property_dict multi_keycg}||iD]}|tr|dr4|d|dgDf|dr||dt t |S)NenumcLg|]!}dt||d"S`` _YAML_MAPrr)r enum_values r0rz,_collect_subschema_types..sB"Cz:>>BBBr/r)rrrrrrrB)rErFproperty_typesrs r0_collect_subschema_typesrPsN"&&y"55 5 5 == ( (   ==  5  ! !&/mmFB&?&?     ]]6 " " 5  ! !)F"3 4 4 4 (( ) ))r/defsct|||dg}t|ts|g}|drd|dD}nq|dr$|t |dn8|dr#|t |dt |dkr |d}n0|td |}|d i}|dg}t|ts|g}t|}t|di|diD]0}t||}|r|d kr| |1|r[t |dkr |d |dS|td d |d} |d | S|pd S)zNReturn a string representing a property type from a given jsonschema. rrHcLg|]!}dt||d"SrJrLrks r0rz&_get_property_type..s?   -. (q!$$ ( ( (   r/rrrrr/items UNDEFINEDz of ())_flatten_schema_refsrrrcrrrPlensortr@rGrWr_get_property_typer) rErQrO property_typerWsub_property_typesprune_undefinedsub_itemsub_typesub_property_docs r0r^r^s---"&&vr22N nd + +*()  P  2?2G      7 # #P6}gNNOOOO   7 # #P6}gNNOOO >a&q)  455500   gr * *E62.. ($ / /201-..O%))GR00%))GR2H2HII,,%h55  x;66 !!(++++8 ! " "a ' '#@@);A)>@@ @$8999>sxx(:;;>>>77%5777  'K'r/c|dz}tjdd|}tjdd||}tjdd||}|S)aParse description from the meta in a format that we can better display in our docs. This parser does three things: - Guarantee that a paragraph will be in a single line - Guarantee that each new paragraph will be aligned with the first paragraph - Proper align lists of items @param description: The original description in the meta. @param prefix: The number of spaces used to align the current description z (\S)\n(\S)z\1 \2z\n\nz\n\n{}z\n( +)-z\n{}-)rsubr<)r#r?list_paragraphs r0_parse_descriptionrismaZN&+>>K&)"2"26":":KHHK&HOON33[K r/src_cfgcHd|vrD|ddd}|||d|vrd|dvrP|dddd}|d||d|dvrY|ddD]J}d|vrD|ddd}|||Kt|dg|dg|dgD]J}d|vrD|ddd}|||KdS) zEFlatten schema: replace $refs in src_cfg with definitions from $defs.z$refz#/$defs/rPrWrrallOfN)poprlupdaterrr)rjrQ reference sub_schemas r0r[r[+s KK''// B?? tI'''' WW% % %(,,V44< >z2 N NI%%d9o666 GR   GR   GR  // Z  "v..66z2FFI   d9o . . .//r/cn|dd}|sdS|D]}||dS)zFlatten schema: Merge allOf. If a schema as allOf, then all of the sub-schemas must hold. Therefore it is safe to merge them. rlN)rmrn)rj sub_schemasrps r0_flatten_schema_all_ofrsEsR ++gt,,K !## z""""##r/ prop_configc,|dd}|di}|di}g}g}|rg|d}|ts||n"|t |t ||D]}|ds|ts/||dda|t |dddt ||}|rd|}|dd kr|dz }|S) aGReturn accumulated property description. Account for the following keys: - top-level description key - any description key present in each subitem under anyOf or allOf Order and deprecated property description after active descriptions. Add a trailing stop "." to any description not ending with ":". r#rPrrrjz.  ri)rrrstriprrrnrrkrG)rt prop_descrrr descriptionsdeprecated_descriptionsrbr#s r0_get_property_descriptionr|Rs33J OOGR ( (E OOGR ( (EL M&&s++ ~.. M    + + + + # * *+? +K+K L L L%''||M**  ||N++     7 > >s C C D D D D # * *$Kh}&=&D&DS&I&IKK    ))E,0GHHIIK'+'' r?c ! ! 3 K r/ c &|dz}g}ddurdSfddD}fd|D}|D]<}|D]#\}} t| |t| | ddur=t | } | d|} |t || t| |t| | | d } | r/t| || d s| d rY|t|| |dz }|t| ||| dgD]} | d s| d rY|t|| |dz }|t| ||d | vsd | vr%|t| ||%>d |S)zDReturn restructured text describing the supported schema properties.r}hiddenTrPc0g|]}dvs |dv|S)rr.rrros r0rz%_get_property_doc..s=  6 ! !Sx0@%@%@ %@%@%@r/)rpatternPropertiesc<g|]}|iSr.)rrrs r0rz%_get_property_doc..s'EEE 3++EEEr/r)r? prop_namer# prop_typerWrr)r?r)rQr?rrQ) rrrWr[rsr|rSCHEMA_PROPERTY_TMPLr<rir^SCHEMA_LIST_ITEM_TMPL_get_property_docrG)rorQr? new_prefixr property_keysproperty_schemas prop_schemaprop_keyrtr#rrW alt_schemas` r0rrys.&JJ zz(t##r6M FEEE}EEE'77 %0%6%6%8%86 6  !Hk d 3 3 3 "; / / /x((D003K@@K OOGX66E   $++!# 2; G G0dCC ,    OOG,,E $UD11199\** eii8K.L.L %%-44#-5 &(J%%)%d:NNN#())GR"8"8J!~~l33 z~~+88 #))188'1U9 #f, "))- *j  ++&+55!!%#$ze6 n ;;z " ""r/metach|d}|sdSt}t|D]\}}tj|dd}|tkr1|dt|dz|d |z }|S)zAReturn restructured text describing the meta examples if present.r%rPr}rrr) rrSCHEMA_EXAMPLES_HEADERrtextwrapindentrinsertSCHEMA_EXAMPLES_SPACER_TEMPLATEr<rG)rr% rst_contentcountexampleindented_liness r0 _get_examplesrsxx ##H r(K#H--11w!&99??EE 0 0 0  ! !299%!)DD    tyy000 r/c|dsdSdd|dD}d|dS)Nr'rPr=c3"K|] }d|dV dS)rKNr.rTs r0rz3_get_activate_by_schema_keys_doc..s; Q r/z**Activate only on keys:** rQ)rrrG)r schema_keyss r0 _get_activate_by_schema_keys_docrsd 88- . .r)) !:;K ; : : ::r/c|t}|r|stdt|}hd}dh}d}||z rd||z }n ||z |z rd||z }|rt |t t|}d|d<|d i}||d r1||d i}tt |} t|| |d <n/#t$r"t d d|d <YnwxYw|d r t|d<t||d<d|d|d<t#jdd|d|d<t'||d<t)jdi|}|S)zReturn reStructured text rendering the provided metadata. @param meta: Dict of metadata to render. @param schema: Optional module schema, if absent, read global schema. @raise KeyError: If metadata lacks an expected key. Nz"Expected non-empty meta and schema>r!r r"r$r%r&r#r'rPz(Missing required keys in module meta: {}z3Additional unexpected keys found in module meta: {}property_header$defsr!)rQ property_docz3Unable to render property_doc due to invalid schemar%r=r$rj-r title_underbarr.)r ValueErrorsetkeysr<KeyErrorrrrrrrAttributeErrorrrSCHEMA_PROPERTY_HEADERrrGrrgrSCHEMA_DOC_TMPL) rror required_keys optional_keys error_message meta_copyrQtemplates r0 get_meta_docrsC~ ?v?=>>> tyy{{  DM//MMt  BII D      - A H H}$    &}%%%Xd^^$$I#%I ::gr " "D xxT $$t*b))dF##'$5f4$H$H$H .!! ''' IJJJ$& .!!!' >'= #$)$//Ij99T)_55Ii"$&sDL"A"AI+K ,,I'(%22 22H Os!D66)E"!E"ctjtjt}t |SrD)r%r5dirnameabspath__file__r) configs_dirs r0 get_modulesrs1'//"'//(";";<  Mr/c|j|j|jg}td|Ddkrt d|jr|jrt dt }|js|jr t|j||j|jd}n|j}td|dS#t$r4}|js"t t|Yd}~dSYd}~dSd}~wt$r'}t t|Yd}~dSd}~wwxYw|jr#tt|jdSdS)z@Handle provided schema args and perform the appropriate actions.cg|]}||Sr.r.)rargs r0rz&handle_schema_args..s 1 1 1CS 1C 1 1 1r/rz;Expected one of --config-file, --system or --docs argumentsz;Invalid flag combination. Cannot use --annotate with --docsNzsystem userdatazValid cloud-config:) config_filersystemr\rrrr:r/rLr+r'r)r argsexclusive_argsrcfg_namer}s r0handle_schema_argsrs& 4;?N 1 1> 1 1 122a77 KLLL }MM KLLL,,K #4;# 3 % +t}   ',+ ' 2 2 2 2 2%   = c!ff           #a&&MMMMMMMMM  # hty!!"""""##s$;B99 D&#C22 D&?D!!D&cft}td|dS)zDTool to validate schema of a cloud-config file or print schema docs.rr)rr parse_argsrs r0mainrs- \\F+V->->-@-@AAA 1r/__main__rD)T)NFFTF)F)r})grYrrloggingr%rsysr collectionsrcollections.abcrcopyr functoolsr itertoolsrr-rr r r r r rr2 cloudinitrrcloudinit.stagesrcloudinit.utilrrrr_r_ValidationErrorr`r getLoggerr(rVERSIONED_USERDATA_SCHEMA_FILErrMr.rrrrrrrrktyping_extensionsrrrrr2r4rZr+rJrrLrgrnrWrvrrrrrrrdrr:r@rBrPr^rir[rsr|rrrrrrrrrrrrrexitr.r/r0rsLL  ######$$$$$$OOOOOOOOOOOOOOOOOO ((((((((!!!!!!AAAAAAAAAA >>>>>>&OO   OOO  %%%g!!!D5' 8 8 &$.LN.">" MMM8888888888888Y8888J     _   .....J...m$ !   #  SM        ( ( ( ( (J ( ( (F ( ( (77hsm7s7777#9    Y .#9 ! ! Y ! ! ! ! R#9 ,$,$ Y ,$,$,$,$^G/G/G/T   t     J#" JJ J J J J  J  JJJJZr,r,r,r,r,r,r,r,t/348  N+  ".1  3:0ggggT    *D*S*T#Y****")(d)($)(3)()()()(Xs,/$/d////4 #D # # # #$4$C$$$$NE#E#dE#$E##E#E#E#E#P  s     ;:;#;;;;;;z;8D>;S;;;;|-T---- 4OOOOOD> $    !!!!H###8    z CHTTVVs,A55A?>A?