Как проверить yml файл на ошибки

Что такое YAML?

YAML (Yet Another Markup Language) или (YAML Ain’t Markup Language) является удобные для человека стандартом сериализации данных для всех языков программирования и часто используется для написания файлов конфигурации.

Об инструменте проверки YAML

Проверить YAML онлайн, также называемая YAML Lint инструментом, была создана для отладки, поиска ошибок и предупреждений в YAML файлах. Этот инструмент позволяет решить проблему путем проверки YAML данных, чтобы его можно было легко отлаживать.

Как это работает?

Просто вставьте свои YAML данные в текстовое поле выше и нажмите кнопку «Проверить», и вы получите результат проверки ниже в следующей форме.

Пример

Правильный YAML:

# test yaml -----------------------------------------------#
namespace: commontests
actor_suffix: Tester
paths:
    tests: tests
    output: tests/_output
    data: tests/_data
    support: tests/_support
settings:
    bootstrap: _bootstrap.php
    colors: true
    memory_limit: 1024M
modules:
    config:
        wtools:
            configFile: 'config/test-local.php'

YAML с ошибкой:

plain
text
is
not
yaml

Check out our primary project!
Browserling

We created Browserling – get a cloud browser in 5 seconds. Check it out!

Pro tips
Master online yaml tools

You can pass input to this tool via

?input

query argument and it will automatically compute output. Here’s how to type it in your browser’s address bar. Click to try!

https://onlineyamltools.com/validate-yaml

?input

=plain%0Atext%0Ais%0Anot%0Ayaml

Coming soon
These yaml tools are on the way

Diff Two YAML Files

Compare two YAML files and show their differences.

Show YAML Statistics

Show detailed information about a YAML config.

Flatten YAML

Quickly flatten a nested YAML config file.

Truncate YAML

Loop over YAML and truncate lists, arrays, and strings.

Create a PHP Array from YAML

Convert YAML config to a PHP data structure.

Convert YAML to Properties

Convert a YAML file to a properties file.

Convert Properties ot YAML

Convert a properties file to a YAML file.

Convert YAML to TOML

Convert a YAML file to a TOML file.

Convert TOML to YAML

Convert a TOML config to a YAML config.

Convert YAML to NestedText

Convert a YAML file to a NestedText file.

Convert NestedText to YAML

Convert a NestedText config to a YAML config.

Convert YAML to PDF

Create a PDF file from a YAML file.

Sort YAML

Sort a YAML file by keys or by values.

Extract Keys from YAML

Find and extract all keys from a YAML file.

Extract Values from YAML

Find and extract all keys from a YAML file.

Extract Strings from YAML

Find and extract all strings from a YAML file.

Extract Numbers from YAML

Find and extract all numbers from a YAML file.

Extract Comments from YAML

Find and extract all comments from a YAML file.

Remove Comments from YAML

Clean up a YAML configuration file from comments.

Add Errors to YAML

Add tiny errors to YAML and create an invalid YAML file.

Change All Quotes to Double Quotes

Normalize all quotes in a YAML file to double quotes.

Change All Quotes to Single Quotes

Normalize all quotes in a YAML file to single quotes.

Visualize YAML Structure

Create an abstract image that represents YAML data.

YAML Validator is easy to use the YAML Validate tool. Copy, Paste, and Validate. This is also called a YAML Lint tool.

What can you do with YAML Validator Online?

  • It helps to Validate your YAML data.
  • It also works as YAML Checker or YAML syntax checker.
  • This tool allows loading the YAML URL to validate. Use your YAML REST URL to validate. Click on the Load URL button, Enter URL and Submit.
  • Users can also validate YAML Files by uploading the file. This functionality also referred to as YAML File validator
  • It helps to save your validated YAML online and Share it on social sites or emails.
  • YAML Validator works well on Windows, MAC, Linux, Chrome, Firefox, Edge, and Safari.
  • This YAML Linter helps a developer who works with JSON data to test and verify.

Example to validate YAML

Valid YAML Try it.

InsuranceCompanies: 
 Time: "May 2021"
 "Top Insurance Companies": 
  - 
   No: 1
   Name: "Berkshire Hathaway ( BRK.A)"
   "Market Capitalization": "$655 billion"
 source: "investopedia.com"
 url: "https://www.investopedia.com/articles/active-trading/111314/top-10-insurance-companies-metrics.asp"                                              

Invalid YAML Try it.

InsuranceCompanies: 
 Time: "May 2021
 "Top Insurance Companies": 
  - 
   No: 1
   Name: "Berkshire Hathaway ( BRK.A)"
   "Market Capitalization": "$655 billion"
 source: "investopedia.com"
 url: "https://www.investopedia.com/articles/active-trading/111314/top-10-insurance-companies-metrics.asp"                                               

For Advanced Users

Lua External URL

Load Lua External URL in Browser URL like this https://codebeautify.org/yaml-validator?url=external-url

https://codebeautify.org/yaml-validator?url=https://gist.githubusercontent.com/cbmgit/b4b3a9361a795e4886966b6e12abeeaf/raw/InsuranceCompanies.yaml

Text Tools

String Utilities

Case Converter

Sort Lines

Diff Checker

Text Editor

JSON Editor

Lorem Ipsum Generator

URL Parser / Query String Splitter

Slug Generator

HTML Stripper

Pastebin

Formatters

HTML Formatter / Minifier

CSS Beautifier / Minifier

JavaScript Beautifier / Minifier

JSON Formatter / Minifier

XML Formatter / Minifier

SQL Formatter / Minifier

Escape and Unescape Tools

HTML Escape / Unescape

JavaScript Escape / Unescape

JSON Escape / Unescape

XML Escape / Unescape

SQL Escape / Unescape

Validators

HTML Validator

JSON Validator

YAML Validator

Credit Card Validator

Regular Expression Tester

Cryptography and Security

Hash Generator

HMAC Generator

Checksum Calculator

Password Generator

Bcrypt Generator / Checker

UUID Generator

Encoders and Decoders

URL Encoder / Decoder

HTML Entity Encoder / Decoder

Base32 Encoder / Decoder

Base58 Encoder / Decoder

Base64 Encoder / Decoder

Base64 Image Encoder / Decoder

JWT Decoder

Converters

JavaScript / JSON Converter

CSV / JSON Converter

JSON / XML Converter

JSON / PHP Array Converter

Text / HEX Converter

Base Converter

Unit Converter

UNIX Timestamp Converter

Image Tools

Color Picker

QR Code Generator

QR Code Reader

EXIF Viewer

Network Tools

My IP

IP Lookup

IP / Number Converter

CIDR / IP Converter

WHOIS Lookup

Background

Recently, we considered an approach, where in a single file document analysts are able to share SIEM (idea) queries, and some form of documentation and or notes. We also needed to make them machine parseable and transformable, in order for us to automate the parts of the queries to feed into a SIEM system.
This sort of idea is not new or ground breaking in anyway. In fact, it is pretty popular in the information security industry to share ideas for threat detection & hunting in YAML, TOML or markdown with code blocks.

YAML and TOML file formats are used a lot in threat detection & hunting rule sharing communities, ever since Sigma — generic signature format for SIEM systems1, came out, I believe. Threat detection & hunting enthusiasts sharing ideas are also making use of YAML file format2.
In fact, vendors like Elastic3 share their detection contents on GitHub as TOML files4.

This allows some form of uniformity in how the contents should be structured and also defines how the machine or automations should extract the information.

If you have the team, time, development and engineering resources, it might be worth looking into just using Sigma and to get contents for different security systems ingesting them automatically.
However, our approach was that, we wanted something to mix SIEM specific queries and some documentations together, while only spending some time into writing a script that can just strip out the query so that it can then be fed into a SIEM system, so we went with our own YAML format.

The downsides of coming up with your own format is that, you need to first define the structure, what field and values are mandatory, what are optional and then make a decision. This downside, however, can be overcome relatively easy in some cases. This was the case for us.

NOTE: Just want the sauce?

The Details

This is how a signature format in YAML looks like:

It has a title, id, description about the rule, the author, references, logsource, detection rules.

Let’s say, for your community or organization, you decided a YAML format inspired by Sigma, however, it is not an extension, and that you do not use a single syntax high-level abstraction for your queries. It looks like:

---
id: 6068c062-627f-4d7c-9250-5059f5417726 # UUIDv4
title: some title for your detection rule
description: a short sentence about the detection rule
references:
  - reference URL 1
  - reference URL 2
analyst_notes: >
  When you see X, you need to check if occurances of A, B, C, D are also there?
  If not, it might indicate a false-positive or a scenario 1 like in Alpha.
  If you see at most 3 out of 4, it is surely suspicious and therefore you should
  look for to find: K, L, M, N.  
query: >
  SELECT * FROM registry WHERE 
  key LIKE 'HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\%%' 
  and name='Debugger';",  
mitre:
  - T1112
jira: PJ-1337

Now you get to a point where these files are stored in Git for version control and some form of automation is in-place, you will need to make sure that the file is properly formatted as YAML and also compliant to your custom schema.
The former can be achieved by using a linting tool like yamllint5. The latter with a library like Yamale6, which is what we went with.

YAML Linting

yamllint is a command line tool and a library you can use in your own tooling. If we run the tool on the YAML file above:

$ yamllint detection_rule.yml
detection_rule.yml
  2:42      warning  too few spaces before comment  (comments)
  11:81     error    line too long (81 > 80 characters)  (line-length)
  15:81     error    line too long (102 > 80 characters)  (line-length)

However, if you would like to combine such a linting process together with other checks your scripts are doing, you can import the library7. Here is an example:

import yamllint
from yamllint.config import YamlLintConfig

raw_yaml = open('detection_rule.yml', 'r').read()

yaml_config = YamlLintConfig("extends: default")

for p in yamllint.linter.run(raw_yaml, yaml_config):
    print(p.desc, p.line, p.rule)

Yamale — schema validation

Yamale is also a command line tool and a library. It comes with a few default validator8 functions, and is also very easily extendable. Here we will see how we could extend it for our schema.

First, we need to come up with a schema dictionary that Yamale can understand to use to validator your YAML files. Let’s consider the following as an initial schema dictionary:

id: str()
title: str()
description: str()
references: list(str())
analyst_notes: str()
query: str()
mitre: list(str())
jira: str()

Running yamale and providing the above schema as follows, yields a validation success.

$ yamale -s schema.yaml detection_rule.yml
Validating /home/user/project-x/detection_rule.yml...
Validation success! 👍

If you kept a close eye, you will have noticed that we initially told yamale that id is a string, however, that is not entirely true.
The validation will also pass if you wrote in a bogus string that is not a UUID. So we will need to extend yamale and write our own validator.

Looking at an example custom validator9 in their example, we can try a proof of concept UUID validator and also include the validation routines as well:

import yamale
import uuid
from yamale.validators import DefaultValidators, Validator

class UUID(Validator):
    """ Custom UUID validator """
    tag = 'uuid'

    def _is_valid(self, value):
        try:
            luuid = uuid.UUID(str(value))
        except ValueError:
            return False
        return True

validators = DefaultValidators.copy()  # This is a dictionary
validators[UUID.tag] = UUID
schema = yamale.make_schema('./schema.yaml', validators=validators)

data = yamale.make_data('./detection_rule.yml')

try:
    yamale.validate(schema, data)
    print('Validation success! 👍')
except ValueError as e:
    print('Validation failed!n%s' % str(e))
    exit(1)

Edit the validator for id in schema.yaml to uuid(), run the script above and it should output:

$ python schema-validate.py
Validation success! 👍

Say, for another custom validator, you want to check and make sure that jira values do confirm to the documented JIRA project key format10, since these are manually entered by analysts. Here we will reuse/sub-class from the built-in Regex validator.

import re
from yamale.validators import Regex

class JIRA(Regex):
    """ Custom JIRA Project ID validator. """
    tag = 'jira'

    def __init__(self, *args, **kwargs):
        self._project_key = str(kwargs.pop('project_key', ''))
        super(JIRA, self).__init__(*args, **kwargs)

        if len(self._project_key) > 0:
            self.regexes = [
                re.compile("^%s-d+$" % (self._project_key))
            ]
        else:
            self.regexes = [
                re.compile("^([A-Z]{2}[0-9]{2})-d+$"),
                re.compile("^([A-Z][A-Z_0-9]+)-d+$"),
            ]

Adjust jira field’s value in schema.yaml as jira(project_key='JP') and on running the script, it should error out:

Validation failed!
Error validating data './detection_rule.yml' with schema './schema.yaml'
	jira: 'PJ-1337' is not a jira match.

Challenge

Let us try writing a schema validator for the Sigma rule we mentioned at the beginning.

---
title: str(min=1, max=256)
id: str()
status: enum('stable', 'testing', 'experimental', required=False)
description: str(required=False)
author: str(required=False)
date: str()
modified: str()
references: list(str(), required=False)
tags: list(str())
logsource: include('logsource')
detection: include('detection')
falsepositives: any(str(), list(), required=False)
level: enum('low', 'medium', 'high', 'critical', required=False)
---
logsource:
  product: str(required=False)
  category: str(required=False)
  service: str(required=False)
  definition: str(required=False)
---
detection:
  selection: any(str(), list(), map(key=str()))
  condition: str()
  timeframe: str(required=False)

Note: At times if you compare Yamale with something like Rx11, the former seems somewhat limiting for the way Sigma was designed. However, I would start with something like Yamale first and then think about Rx later on.

Full Example Files

detection_rule.yml

---
id: 6068c062-627f-4d7c-9250-5059f5417726
title: some title for your detection rule
description: a short sentence about the detection rule
references:
  - reference URL 1
  - reference URL 2
analyst_notes: >
  When you see X, you need to check if occurances of A, B, C, D are also there?
  If not, it might indicate a false-positive or a scenario 1 like in Alpha.
  If you see at most 3 out of 4, it is surely suspicious and therefore you should
  look for to find: K, L, M, N.  
query:
  SELECT * FROM registry WHERE 
  key LIKE 'HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\%%' 
  and name='Debugger';",
mitre:
  - T1112
jira: PJ-1337

schema.yaml

id: uuid()
title: str()
description: str()
references: list(str())
analyst_notes: str()
query: str()
mitre: list(str())
jira: jira()

validator.py

import yamale
import uuid
import re
from yamale.validators import DefaultValidators, Validator, Regex

class UUID(Validator):
    """ Custom UUID validator """
    tag = 'uuid'

    def __init__(self, *args, **kwargs):
        super(UUID, self).__init__(*args, **kwargs)
        self._version = int(kwargs.pop('version', 4))

    def _is_valid(self, value):
        try:
            luuid = uuid.UUID(str(value), version=self._version)
        except ValueError:
            return False
        return True

class JIRA(Regex):
    """ Custom JIRA Project ID validator. """
    tag = 'jira'

    def __init__(self, *args, **kwargs):
        self._project_key = str(kwargs.pop('project_key', ''))
        super(JIRA, self).__init__(*args, **kwargs)

        if len(self._project_key) > 0:
            self.regexes = [
                re.compile("^%s-d+$" % (self._project_key))
            ]
        else:
            self.regexes = [
                re.compile("^([A-Z]{2}[0-9]{2})-d+$"),
                re.compile("^([A-Z][A-Z_0-9]+)-d+$"),
            ]

validators = DefaultValidators.copy()  # This is a dictionary
validators[UUID.tag] = UUID
validators[JIRA.tag] = JIRA
schema = yamale.make_schema('./schema.yaml', validators=validators)

data = yamale.make_data('./detection_rule.yml')

try:
    yamale.validate(schema, data)
    print('Validation success! 👍')
except ValueError as e:
    print('Validation failed!n%s' % str(e))
    exit(1)

Понравилась статья? Поделить с друзьями:

Не пропустите эти материалы по теме:

  • Яндекс еда ошибка привязки карты
  • Как проверить xml файл на ошибки
  • Как проверить xiaomi на ошибки
  • Как проверить wot на наличие ошибок
  • Как проверить wordpress на ошибки

  • 0 0 голоса
    Рейтинг статьи
    Подписаться
    Уведомить о
    guest

    0 комментариев
    Старые
    Новые Популярные
    Межтекстовые Отзывы
    Посмотреть все комментарии