CFE-3041: Made testall fail regular acceptance tests which log errors by aleksandrychev · Pull Request #6405 · cfengine/core

aleksandrychev · 2026-10-14T12:25:35Z

Tests that print Pass message but have logs errors will fail with FAIL (Error printed) message. Tests which are expected to log errors should use .error.cf or .error.pass.cf suffix. see: tests/acceptance/README

Ticket: CFE-3041

Tests that print Pass message but have logs errors will fail with `FAIL (Error printed)` message. Tests which are expected to log errors should use `.error.cf` or `.error.pass.cf` suffix. see: `tests/acceptance/README` Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

aleksandrychev · 2026-10-14T12:25:57Z

@cf-bottom jenkins please

cf-bottom · 2026-10-14T12:31:42Z

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/14799/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14799/

olehermanse

This cannot be done on its own - there are hundreds of tests that need to be reviewed and adjusted after this change in testall.

aleksandrychev · 2026-10-14T18:43:22Z

This cannot be done on its own - there are hundreds of tests that need to be reviewed and adjusted after this change in testall.

I see, there are quite a lot of them. So, we need to identify which ones are expected and should be renamed, and also find the ones that shouldn't be there. As a first step, I'll create a table listing each failure and the errors printed.

aleksandrychev · 2026-10-15T11:07:45Z

Acceptance tests failing with "Error printed"

#	File	Printed error
1	`00_basics/01_compiler/string_contexts.cf`	`error: class expressions can't be separated by whitespace without an intervening operator in expression 'cf engine'`
2	`00_basics/03_bodies/invalid_action_policy_errors.cf`	`error: Fatal CFEngine error: Unrecognized 'action_policy' value 'waarn''`
3	`00_basics/03_bodies/report_error_when_body_missing.cf`	`error: Policy failed validation with command '"$WORKDIR/bin/cf-promises" -c "00_basics/03_bodies/report_error_when_body_missing.cf.sub"'` `error: CFEngine was not able to get confirmation of promises from cf-promises, so going to failsafe` `error: CFEngine failsafe.cf: $WORKDIR/inputs $WORKDIR/inputs/failsafe.cf`
4	`00_basics/04_bundles/wrong-bundle-type.cf`	`error: There are syntax errors in policy files` `error: Policy failed validation with command '"$WORKDIR/bin/cf-promises" -c "00_basics/04_bundles/wrong-bundle-type.sub.x.cf"'` `error: CFEngine was not able to get confirmation of promises from cf-promises, so going to failsafe` `error: CFEngine failsafe.cf: $WORKDIR/inputs $WORKDIR/inputs/failsafe.cf`
5	`00_basics/06_host_specific_data/10-variable-references.cf`	`error: Failed to load CMDB data`
6	`00_basics/CFE-1886.cf`	`error: Could not open JSON file ./00_basics/def.json`
7	`00_basics/def.json/bad-json.cf`	`error: Could not parse JSON file $WORKDIR/inputs/def.json: Unable to parse json data as object, unrecognized token beginning entry'`
8	`00_basics/def.json/json-inputs-not-array.cf`	`error: Unresolved variable '@(def.augments_inputs)' in input list, cannot parse'`
9	`00_basics/linesep.cf`	`error: Could not open JSON file ./00_basics/def.json`
10	`00_basics/namespaces_can_not_contain_funny_characters.cf`	`error: Could not open JSON file ./00_basics/def.json` `error: Could not open JSON file 00_basics/def.json` `error: There are syntax errors in policy files` `error: Policy failed validation with command '"$WORKDIR/bin/cf-promises" -c "00_basics/namespaces_can_not_contain_funny_characters.cf.sub"'` `error: CFEngine was not able to get confirmation of promises from cf-promises, so going to failsafe` `error: CFEngine failsafe.cf: $WORKDIR/inputs $WORKDIR/inputs/failsafe.cf`
11	`01_vars/01_basic/013.cf`	`error: Proposed executable file '/nonexisting' doesn't exist` `error: execresult '/nonexisting' is assumed to be executable but isn't`
12	`01_vars/01_basic/can_not_define_variables_in_remote_bundles.cf`	`error: Ignoring remotely-injected variable 'variable'` `error: Remote bundle variable injection detected!` `error: Variable identifier 'check.variable' is not legal` `error: Promise belongs to bundle 'test' in file './01_vars/01_basic/can_not_define_variables_in_remote_bundles.cf' near line 17` `error: Comment is 'This should not be allowed, perhaps the first part`
13	`01_vars/01_basic/this_promiser_with_readjson.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.cf_promises_validated': No such file or directory`
14	`01_vars/02_functions/006.cf`	`error: Couldn't get filesystem info for '/lkjqeroiu'. (statfs: No such file or directory)`
15	`01_vars/02_functions/011.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
16	`01_vars/02_functions/012.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
17	`01_vars/02_functions/013.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
18	`01_vars/02_functions/014.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
19	`01_vars/02_functions/015.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
20	`01_vars/02_functions/016.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
21	`01_vars/02_functions/017.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
22	`01_vars/02_functions/018.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
23	`01_vars/02_functions/019.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
24	`01_vars/02_functions/020.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
25	`01_vars/02_functions/021.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
26	`01_vars/02_functions/024.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
27	`01_vars/02_functions/025.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
28	`01_vars/02_functions/026.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
29	`01_vars/02_functions/027.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
30	`01_vars/02_functions/028.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)` `error: Anomalous ending '.4' while parsing integer number: 123.456` `error: Presumed int value '123.456' read from file '$WORKDIR/tmp/TEST.cfengine' has no recognizable value`
31	`01_vars/02_functions/029.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
32	`01_vars/02_functions/030.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)` `error: Anomalous ending '.4' while parsing integer number: 123.456` `error: Presumed int value '123.456' read from file '$WORKDIR/tmp/TEST.cfengine' has no recognizable value`
33	`01_vars/02_functions/031.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
34	`01_vars/02_functions/032.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)` `error: Anomalous ending '.4' while parsing integer number: 123.456` `error: Presumed int value '123.456' read from file '$WORKDIR/tmp/TEST.cfengine' has no recognizable value`
35	`01_vars/02_functions/033.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
36	`01_vars/02_functions/034.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
37	`01_vars/02_functions/037.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
38	`01_vars/02_functions/038.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
39	`01_vars/02_functions/039.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
40	`01_vars/02_functions/040.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
41	`01_vars/02_functions/041.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
42	`01_vars/02_functions/042.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
43	`01_vars/02_functions/043.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
44	`01_vars/02_functions/044.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
45	`01_vars/02_functions/045.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
46	`01_vars/02_functions/046.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
47	`01_vars/02_functions/047.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
48	`01_vars/02_functions/050.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
49	`01_vars/02_functions/400.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
50	`01_vars/02_functions/401.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
51	`01_vars/02_functions/410.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
52	`01_vars/02_functions/420.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
53	`01_vars/02_functions/421.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
54	`01_vars/02_functions/440.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
55	`01_vars/02_functions/441.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
56	`01_vars/02_functions/442.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
57	`01_vars/02_functions/443.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
58	`01_vars/02_functions/444.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
59	`01_vars/02_functions/445.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
60	`01_vars/02_functions/447.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
61	`01_vars/02_functions/450.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
62	`01_vars/02_functions/500.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
63	`01_vars/02_functions/501.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
64	`01_vars/02_functions/520.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
65	`01_vars/02_functions/521.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
66	`01_vars/02_functions/600.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
67	`01_vars/02_functions/601.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
68	`01_vars/02_functions/620.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
69	`01_vars/02_functions/621.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
70	`01_vars/02_functions/CFE-2704-0.cf`	`error: data[env] is not mergeable as it it not a container`
71	`01_vars/02_functions/CFE-2704-1.cf`	`error: data[env] is not mergeable as it it not a container`
72	`01_vars/02_functions/classfilterdata_collecting.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
73	`01_vars/02_functions/countlinesmatching_returns_0_on_inaccessable_file.cf`	`error: File '/asd/fgh/qwertyio0p' could not be read in countlinesmatching(). (fopen: No such file or directory)`
74	`01_vars/02_functions/execresult_action_immediate.cf`	`error: Apparent bundle 'immediate' was undeclared, but used in a promise near line 15 of ./01_vars/02_functions/execresult_action_immediate.cf (possible unquoted literal value)` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/execresult_action_immediate_report'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/execresult_action_immediate_report' (stat: No such file or directory)`
75	`01_vars/02_functions/findfiles_up.cf`	`error: Function findfiles_up, path '$WORKDIR/tmp/TESTDIR.cfengine/core/libntech/libutils/' is not a directory`
76	`01_vars/02_functions/getfields.cf`	`error: File '$WORKDIR/tmp/TEST.cfengine' could not be read in getfields(). (fopen: No such file or directory)`
77	`01_vars/02_functions/ifelse.cf`	`error: class expressions can't be separated by whitespace without an intervening operator in expression 'this is not true'` `error: class expressions can't be separated by whitespace without an intervening operator in expression 'this is also not true'`
78	`01_vars/02_functions/long_string_from_text_file.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.data'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.data' (stat: No such file or directory)`
79	`01_vars/02_functions/lsdir.cf`	`error: Directory '$WORKDIR/tmp/TESTDIR.cfengine/b' could not be accessed in lsdir(), (opendir: No such file or directory)` `error: Directory '$WORKDIR/tmp/TESTDIR.cfengine' could not be accessed in lsdir(), (opendir: No such file or directory)`
80	`01_vars/02_functions/nth_datacontainer.cf`	`error: Failed to parse integer number: a` `error: Failed to parse integer number: b` `error: Failed to parse integer number: c` `error: Failed to parse integer number: d` `error: Failed to parse integer number: foo` `error: Failed to parse integer number: bar` `error: Failed to parse integer number:` `error: Failed to parse integer number: string_2`
81	`01_vars/02_functions/parsestringarray_dontread_comment_duplicate_lastline2.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
82	`01_vars/02_functions/parsestringarray_weird_indices_real_comments_noemptyfields2.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
83	`01_vars/02_functions/peers.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.nohosts'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.nopeers'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.somepeers'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.allpeers'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.nohosts' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.nopeers' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.somepeers' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.allpeers' (stat: No such file or directory)`
84	`01_vars/02_functions/read-file-endless-loop.cf`	`error: Unable to read data from file '/'. (fgets: Is a directory)` `error: In function 'regline', error reading from file. (getline: Is a directory)`
85	`01_vars/02_functions/read_empty_file.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
86	`01_vars/02_functions/readdata.cf`	`error: readdata: error parsing JSON file '01_vars/02_functions/readdata.cf.csv.guess': Unable to parse json data as object, unrecognized token beginning entry` `error: readdata: error parsing JSON file '01_vars/02_functions/readdata.cf': Unable to parse json data as object, unrecognized token beginning entry`
87	`01_vars/02_functions/readdata_yaml.cf`	`error: readdata: error parsing JSON file '01_vars/02_functions/readdata_yaml.cf.yaml.guess': Unable to parse json data as number, - not at the start or not after exponent` `error: readdata: non-container from parsing YAML file '01_vars/02_functions/readdata_yaml.cf'`
88	`01_vars/02_functions/readfile_absent_does_not_error_in_pre_eval.cf`	`error: Could not examine file '/tmp/supercalifragilisticexpialidocious'` `error: CfReadFile: Could not examine file '/tmp/supercalifragilisticexpialidocious' (stat: No such file or directory)`
89	`01_vars/02_functions/readstringarray_dontread_comment_duplicatekey_lastline.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
90	`01_vars/02_functions/readstringarrayidx_order.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.orig.txt'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.orig.txt' (stat: No such file or directory)`
91	`01_vars/02_functions/regextract.cf`	`error: Failed to parse real number: $(init.pair[2])`
92	`01_vars/04_containers/mergecontainer_conflict.cf`	`error: Error parsing JSON expression ': "y", "third"]': Unable to parse json data as object, unrecognized token beginning entry`
93	`01_vars/04_containers/parseyaml_abuse.cf`	`error: Non-container from parsing YAML expression 'xyzzz'`
94	`01_vars/04_containers/readjson.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
95	`01_vars/04_containers/readjson_arrays.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
96	`01_vars/04_containers/readjson_new.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
97	`01_vars/04_containers/readjson_null.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory` `error: readjson: non-container from parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json'`
98	`01_vars/04_containers/readjson_objects.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
99	`01_vars/04_containers/readjson_scalar.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory` `error: readjson: non-container from parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json'`
100	`01_vars/04_containers/readjson_subindex.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
101	`01_vars/04_containers/readjson_subindex_quoted.cf`	`error: readjson: data error parsing JSON file '$WORKDIR/tmp/TEST.cfengine.json': No such file or directory`
102	`02_classes/01_basic/cancel_attribute_modifier_mutex.cf`	`error: The 'cancel' attribute cannot be combined with 'persistence', 'scope', or 'timer_policy' (class 'c_scope')` `error: The 'cancel' attribute cannot be combined with 'persistence', 'scope', or 'timer_policy' (class 'c_timer')` `error: The 'cancel' attribute cannot be combined with 'persistence', 'scope', or 'timer_policy' (class 'c_persist')` `error: The 'cancel' attribute cannot be combined with 'persistence', 'scope', or 'timer_policy' (class 'c_timer')'`
103	`02_classes/01_basic/cancel_attribute_mutex.cf`	`error: Irreconcilable constraints in classes for 'c_and'` `error: Irreconcilable constraints in classes for 'c_or'` `error: Irreconcilable constraints in classes for 'c_xor'` `error: Irreconcilable constraints in classes for 'c_not'` `error: Irreconcilable constraints in classes for 'c_dist'` `error: Irreconcilable constraints in classes for 'c_select'` `error: Irreconcilable constraints in classes for 'c_expression'` `error: Irreconcilable constraints in classes for 'c_select''`
104	`02_classes/01_basic/cancel_attribute_persistent_lmdb.cf`	`error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/cancel_lmdb_before'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/cancel_lmdb_after'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/cancel_lmdb_before' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/cancel_lmdb_after' (stat: No such file or directory)`
105	`02_classes/01_basic/common_bundle_select_class_empty_list.cf`	`error: No classes to select on RHS` `error: Promise belongs to bundle 'test' in file './02_classes/01_basic/common_bundle_select_class_empty_list.cf' near line 24`
106	`02_classes/01_basic/common_bundle_select_class_variable_expands_empty_list.cf`	`error: No classes to select on RHS` `error: Promise belongs to bundle 'test' in file './02_classes/01_basic/common_bundle_select_class_variable_expands_empty_list.cf' near line 21`
107	`02_classes/01_basic/persistent_timer_policy_default_lmdb.cf`	`error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_policy'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_expires1'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_expires2'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_policy' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_expires1' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/lmdb_expires2' (stat: No such file or directory)`
108	`02_classes/02_functions/classesmatching_inherit_2.cf`	`error: Could not examine file '02_classes/02_functions/defined_classes.log'` `error: CfReadFile: Could not examine file '02_classes/02_functions/defined_classes.log' (stat: No such file or directory)`
109	`02_classes/02_functions/isipinsubnet-valid-ip.cf`	`error: Invalid reference IPv4: not-an-ip-address`
110	`05_processes/01_matching/process_select_fncalls.cf`	`error: Could not get UID for user 'nosuchuser', (getpwnam: not found)`
111	`08_commands/arglist.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.actual'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.actual' (stat: No such file or directory)`
112	`10_files/02_maintain/changes_silent.cf`	`error: File '$WORKDIR/state/file_changes.log' could not be read in countlinesmatching(). (fopen: No such file or directory)` `error: File '$WORKDIR/tmp/TESTDIR.cfengine/changes_silent.out' could not be read in countlinesmatching(). (fopen: No such file or directory)`
113	`10_files/09_insert_lines/multi-files-dirs.cf`	`error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirA/testfa'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirB/testfb'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirC/testfc'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirA/testfa' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirB/testfb' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/dirC/testfc' (stat: No such file or directory)`
114	`10_files/09_insert_lines/preserve_block_blank_lines.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.test'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.test' (stat: No such file or directory)`
115	`10_files/templating/empty_template_empty_file.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
116	`10_files/templating/large_template_blocks.cf`	`error: File '$WORKDIR/tmp/TEST.cfengine' could not be read in countlinesmatching(). (fopen: No such file or directory)`
117	`10_files/templating/mustache_abuse.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.0'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.1'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.2'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.3'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.4'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.5'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.6'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.0' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.1' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.2' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.3' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.4' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.5' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.6' (stat: No such file or directory)`
118	`10_files/templating/mustache_container_serialization.cf`	`error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/a'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/b'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/c'` `error: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/d'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/a' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/b' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/c' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TESTDIR.cfengine/d' (stat: No such file or directory)`
119	`10_files/templating/mustache_datastate_demo.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
120	`10_files/templating/mustache_demo.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
121	`10_files/templating/mustache_expect_list_find_string.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.good'` `error: Could not examine file '$WORKDIR/tmp/TEST.cfengine.bad'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.good' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine.bad' (stat: No such file or directory)` `error: Error rendering mustache template '10_files/templating/mustache_expect_list_find_string.cf.mustache'` `error: Errors encountered when actuating files promise '$WORKDIR/tmp/TEST.cfengine.bad'` `error: Method 'file_mustache_jsonstring' failed in some repairs` `error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs`
122	`10_files/templating/mustache_html_escape.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
123	`10_files/templating/mustache_inline_demo.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
124	`10_files/templating/mustache_render_quote.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
125	`10_files/templating/mustache_repair_existing.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`
126	`10_files/zendesk_1776/zendesk_1776.cf`	`error: Directory '$WORKDIR/tmp/TESTDIR.cfengine/file_select/' could not be accessed in lsdir(), (opendir: No such file or directory)`
127	`15_control/01_common/classic_evaluation_custom_promise_types.cf`	`error: Could not examine file '/tmp/dummyfile'` `error: CfReadFile: Could not examine file '/tmp/dummyfile' (stat: No such file or directory)` `error: Promise module '15_control/01_common/top_down/dummy_promise_type.py' does not exist`
128	`15_control/01_common/classic_evaluation_long_policy.cf`	`error: Could not examine file '/tmp/example'` `error: Could not examine file '/tmp/otherfile'` `error: CfReadFile: Could not examine file '/tmp/example' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '/tmp/otherfile' (stat: No such file or directory)`
129	`15_control/01_common/top_down_evaluation_long_policy.cf`	`error: Could not examine file '/tmp/example'` `error: Could not examine file '/tmp/otherfile'` `error: CfReadFile: Could not examine file '/tmp/example' (stat: No such file or directory)` `error: CfReadFile: Could not examine file '/tmp/otherfile' (stat: No such file or directory)`
130	`15_control/02_agent/abortbundle_regex.cf`	`error: Couldn't find any class matching 'abort_bundle.*'` `error: Bundle 'test' aborted on defined class 'abort_bundle_please'`
131	`29_simulate_mode/simulate_safe_functions.cf`	`error: Could not examine file '$WORKDIR/state/agent.log'` `error: CfReadFile: Could not examine file '$WORKDIR/state/agent.log' (stat: No such file or directory)` `error: Finished command related to promiser '"$WORKDIR/bin/cf-agent" --timestamp --debug --simulate=manifest -Kf 29_simulate_mode/simulate_safe_functions.cf.sub >$WORKDIR/state/agent.log' -- an error occurred, returned 1` `error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs`
132	`30_custom_promise_types/17_early_module_failure.cf`	`error: Did not receive header from promise module '30_custom_promise_types/bad_import.py'` `error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs'`
133	`30_custom_promise_types/23_action_policy/dryrun_supported.cf`	`error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs'`
134	`30_custom_promise_types/23_action_policy/dryrun_unsupported.cf`	`error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs'`
135	`30_custom_promise_types/23_action_policy/explicit_warn_supported.cf`	`error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs'`
136	`30_custom_promise_types/23_action_policy/explicit_warn_unsupported.cf`	`error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs'`
137	`30_custom_promise_types/23_action_policy/simulate_supported.cf`	`error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs`
138	`30_custom_promise_types/23_action_policy/simulate_unsupported.cf`	`error: Not making changes to the system, but the custom promise module '30_custom_promise_types/23_action_policy/example_module_without_action_policy.sh' doesn't support action_policy` `error: Method 'test' failed in some repairs` `error: Method 'test_run' failed in some repairs`
139	`31_tickets/CFE-3866/test.cf`	`error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'` `error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)`

aleksandrychev · 2026-10-15T11:17:58Z

at least half of errors can be fixed easily with a fileexists guard.
for example 01_vars/02_functions/011.cf

error: Could not examine file '$WORKDIR/tmp/TEST.cfengine'
error: CfReadFile: Could not examine file '$WORKDIR/tmp/TEST.cfengine' (stat: No such file or directory)

the error comes from the test bundle:

vars:
    "nums" slist => readstringlist("$(G.testfile)", "NoComment", "\s+", 5, 100);

vars are evaluated early (during policy validation and pre-evaluation), before init has created the file, so the read function fails and logs an error. A later pass reads the file fine, so the test still passes.

it can be fixed by adding a fileexists guard.

These tests read files created by their own init bundle, but pre-evaluation calls the read functions before the files exist, logging "Could not examine file" errors which now fail regular tests. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Errors printed by the command were reported on lines starting with "error:", so testall mistook them for errors of the test itself. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

These tests read files and directories created by their own policy, but pre-evaluation calls the functions before they exist, logging errors which now fail regular tests. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The agent loads def.json from the directory of the input policy file, so tests in 00_basics/ tried to read this directory as their augments file and logged "Could not open JSON file" errors. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The test used the 'immediate' body from plucked.sub.cf before it was loaded during pre-evaluation, so the promise is now skipped until the test framework is loaded. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The test referred to a promise module which never existed and did not run its init and cleanup bundles. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

The check bundle counted lines in files which do not exist until the test bundle has run, but the agent evaluates the functions of all promises before any bundle runs. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

wrong-bundle-type.cf printed the sub-agent output in its DEBUG report, so the sub-agent errors were mistaken for errors of the test, and ifelse.cf used invalid class expressions as false conditions. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

These tests intentionally exercise error paths, the reason is documented in the header of each test. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

aleksandrychev requested review from craigcomstock and olehermanse October 14, 2026 12:25

olehermanse requested changes Oct 14, 2026

View reviewed changes

aleksandrychev marked this pull request as draft October 14, 2026 18:43

aleksandrychev added 10 commits October 15, 2026 14:21

Indented command output in dcs_passif_output() DEBUG report

8716a17

Errors printed by the command were reported on lines starting with "error:", so testall mistook them for errors of the test itself. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Fixed classic_evaluation_custom_promise_types.cf logging errors

bef57e9

The test referred to a promise module which never existed and did not run its init and cleanup bundles. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Renamed acceptance tests which log errors on purpose to .error.pass.cf

089e2a1

These tests intentionally exercise error paths, the reason is documented in the header of each test. Ticket: CFE-3041 Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Fixed linting issues

7ef4940

Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>

Navigation Menu

Repository navigation

CFE-3041: Made testall fail regular acceptance tests which log errors - #6405

CFE-3041: Made testall fail regular acceptance tests which log errors#6405
aleksandrychev wants to merge 11 commits into
cfengine:masterfrom
aleksandrychev:CFE-3041

aleksandrychev commented Oct 14, 2026

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

cf-bottom commented Oct 14, 2026

Uh oh!

olehermanse left a comment

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

aleksandrychev commented Oct 15, 2026

Uh oh!

aleksandrychev commented Oct 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Conversation

aleksandrychev commented Oct 14, 2026

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

cf-bottom commented Oct 14, 2026

Uh oh!

olehermanse left a comment

Choose a reason for hiding this comment

Uh oh!

aleksandrychev commented Oct 14, 2026

Uh oh!

aleksandrychev commented Oct 15, 2026

Acceptance tests failing with "Error printed"

Uh oh!

aleksandrychev commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

aleksandrychev commented Oct 15, 2026 •

edited

Loading