Problems with YAML

[kobrir]

After the recent fix parsing YAML into the glz::generic started to compile, but there is still problems with it, I have made some test cases:

#include <string>
#include <print>
#include <glaze/glaze.hpp>
#include <glaze/yaml.hpp>

void testYAML(std::string yaml)
{
  std::print(R"(====================Source:
{}
====================Result:
)", yaml);

  glz::generic parsed;


  auto ec=glz::read_yaml<glz::opts{.error_on_unknown_keys = false}>(parsed, yaml);
  if(ec)
    std::print("Error: {}", glz::format_error(ec, yaml));
  else
  {
    std::string yaml2;
    void(glz::write_yaml(parsed, yaml2));
    std::print("{}", yaml2);
  }
  std::print("\n");
}

int main(int nargs,char *args[])
{
  testYAML(R"({"name": "Alice", "age": 30})");

  testYAML(R"({"name": "Alice",
"age": 30})");

  testYAML("---\nname: Alice");

  return 0;
}

From this program I am getting this output:

====================Source:
{"name": "Alice", "age": 30}
====================Result:
age:
30name:
Alice
====================Source:
{"name": "Alice",
"age": 30}
====================Result:
Error: 1:18: syntax_error
   {"name": "Alice",
"age": 30}
                    ^
====================Source:
---
name: Alice
====================Result:
name

[stephenberry]

I merged in a ton of YAML fixes and improvements in #2293. Many more unit tests for glz::generic and YAML handling as well. This should fix your issue, let me know if you run into any other issues. Thanks for the error report!

[kobrir]

It got better, but I am still getting incorrect results with more complex files after the roundtrip with read_yaml and write_yaml, for example with this:

---
name: Shell-Unix-Generic
hidden: true
scope: source.shell
contexts:
  prototype:
    - include: scope:source.shell.bash#prototype
  main:
    - include: scope:source.shell.bash

I get this:

contexts:
main:
- {include: 'scope:source.shell.bash'}

prototype:
- {include: 'scope:source.shell.bash'}


hidden: true
name: Shell-Unix-Generic
scope: source.shell

And for this file I am getting a syntax error from read_yaml:
https://github.com/sublimehq/Packages/blob/master/Python/Python.sublime-syntax

[stephenberry]

Thanks for the feedback. A challenge with YAML is that there are multiple ways to format the same data, and so round-tripping isn't really possible without the user configuring some options. But, I'll try to set defaults to what would be common and then provide ways to customize output format.

And, I'll look into the syntax error you're getting on reading.

[kobrir]

Some customizations would be good, but I, personally, don't mind the format change. My main concerns here are the missing indentations and that the parser seems to mistake #prototype for a comment.

And thanks for working on this! Glaze is already a great library, with the YAML support it would be even more useful.

[stephenberry]

Thanks! I'll have some more fixes soon. The parsing issue was that Glaze wasn't handling the % directive. And, yes there's a bug with # handling.

I'm also considering changing the underlying map in glz::generic to a flat_map, because right now it uses a std::map, but this causes keys to be sorted in lexicographical order, rather than maintaining the order of the input, which is what I think most users want for the ability to roundtrip.

[stephenberry]

The immediate issues with % and # have been fixed in #2297

[stephenberry]

I made a silly comment about using flat_map for order, because it also doesn't preserve order. I think we just want an order preserving map for the sake of round tripping yaml.

[kobrir]

The first example is working fine now, but the parser is still having problems with the long file, first it stumbles on the comment here:

match: ^|(?=\S)  # Note: Ensure to highlight shebang if Python is embedded.

And if the comment is removed it then starts to stumble on the multiline strings like this:

scope:
  meta.statement.conditional.case.python
  punctuation.section.block.begin.python

As for preserving the order - I am not sure it is needed here, after all some information would be lost anyway (like the comments).

[kobrir]

Another issue is that the parser seems to treat strings starting with . as numbers and fails to parse them:

file_extensions:
 - .c
 - .cpp

[kobrir]

And some more issues:

#include <concepts>
#include <string>
#include <print>
#include <glaze/glaze.hpp>
#include <glaze/yaml.hpp>


struct ConfigFile
{
  std::vector<int> servers;
};

template<class T>
void testYAML(std::string yaml)
{
  std::print(R"(====================Source:
{}
====================Result:
)", yaml);

  T parsed;


  auto ec=glz::read_yaml<glz::opts{.error_on_unknown_keys = false}>(parsed, yaml);
  if(ec)
    std::print("Error: {}", glz::format_error(ec, yaml));
  else
  {
    std::string yaml2;
    if constexpr (std::same_as<T, glz::generic>)
      void(glz::write_yaml(parsed, yaml2));
    else
      void(glz::write_json(parsed, yaml2));
    std::print("{}", yaml2);
  }
  std::print("\n");
}


int main(int nargs,char *args[])
{
  testYAML<glz::generic>(R"({
  "servers":
  [

  ]
}
)");

  testYAML<ConfigFile>(R"({
  "servers":
  [

  ]
}
)");

  testYAML<ConfigFile>(R"({"servers":[]})");

  return 0;
}

Here the first string parsed correctly, but then the empty array was stringified as just an empty space
Second test produces a syntax error
Third seems fine, but note that I am calling write_yaml or write_json depending on the type T in testYAML, I have tried to use write_yaml unconditionally there but got a compilation error from clang-cl:

In file included from 1.cpp:5:
In file included from glaze/include\glaze/yaml.hpp:9:
glaze/include\glaze/yaml/write.hpp(707,72): error: no member named 'values' in 'glz::reflect<ConfigFile>'
  707 |                   auto&& member = get_member(value, get<I>(reflect<V>::values));
      |                                                            ~~~~~~~~~~~~^
glaze/include\glaze/yaml/write.hpp(697,38): note: while substituting into a lambda expression here
  697 |          for_each<N>([&]<size_t I>() {
      |                                      ^
glaze/include\glaze/yaml/write.hpp(904,19): note: in instantiation of function template specialization 'glz::yaml::write_block_mapping<yaml::yaml_opts{450, true, true, 2, false, 0}, ConfigFile &, std::basic_string<char> &, glz::context &, unsigned long long>' requested here
  904 |             yaml::write_block_mapping<Opts>(value, ctx, b, ix, indent_level);
      |                   ^
glaze/include\glaze/core/write.hpp(28,57): note: in instantiation of function template specialization 'glz::to<450, ConfigFile>::op<yaml::yaml_opts{450, true, true, 2, false, 0}, std::basic_string<char> &, ConfigFile &, glz::context &, unsigned long long>' requested here
   28 |       to<Opts.format, std::remove_cvref_t<T>>::template op<Opts>(std::forward<T>(value), ctx, buffer, ix);
      |                                                         ^
glaze/include\glaze/core/write.hpp(80,14): note: in instantiation of function template specialization 'glz::write<yaml::yaml_opts{450, true, true, 2, false, 0}, ConfigFile &, std::basic_string<char>, glz::context &>' requested here
   80 |       return write<Opts>(std::forward<T>(value), buffer, ctx);
      |              ^
glaze/include\glaze/yaml/write.hpp(973,14): note: in instantiation of function template specialization 'glz::write<yaml::yaml_opts{450, true, true, 2, false, 0}, ConfigFile &, std::basic_string<char>>' requested here
  973 |       return write<set_yaml<Opts>()>(std::forward<T>(value), buffer);
      |              ^
1.cpp(31,17): note: in instantiation of function template specialization 'glz::write_yaml<yaml::yaml_opts{450, true, true, 2, false, 0}, ConfigFile &, std::basic_string<char>>' requested here
   31 |       void(glz::write_yaml(parsed, yaml2));
      |                 ^
1.cpp(50,3): note: in instantiation of function template specialization 'testYAML<ConfigFile>' requested here
   50 |   testYAML<ConfigFile>(R"({
      |   ^
In file included from 1.cpp:5:
In file included from glaze/include\glaze/yaml.hpp:9:
glaze/include\glaze/yaml/write.hpp(728,74): error: no member named 'values' in 'glz::reflect<ConfigFile>'
  728 |             decltype(auto) member = get_member(value, get<I>(reflect<V>::values));
      |                                                              ~~~~~~~~~~~~^