Что такое YAML?
YAML (Yet Another Markup Language) или (YAML Ain’t Markup Language) является удобные для человека стандартом сериализации данных для всех языков программирования и часто используется для написания файлов конфигурации.
Об инструменте проверки YAML
Проверить YAML онлайн, также называемая YAML Lint инструментом, была создана для отладки, поиска ошибок и предупреждений в YAML файлах. Этот инструмент позволяет решить проблему путем проверки YAML данных, чтобы его можно было легко отлаживать.
Как это работает?
Просто вставьте свои YAML данные в текстовое поле выше и нажмите кнопку «Проверить», и вы получите результат проверки ниже в следующей форме.
Пример
Правильный YAML:
# test yaml -----------------------------------------------#
namespace: commontests
actor_suffix: Tester
paths:
tests: tests
output: tests/_output
data: tests/_data
support: tests/_support
settings:
bootstrap: _bootstrap.php
colors: true
memory_limit: 1024M
modules:
config:
wtools:
configFile: 'config/test-local.php'
YAML с ошибкой:
plain text is not yaml
Check out our primary project!
Browserling
We created Browserling – get a cloud browser in 5 seconds. Check it out!
Pro tips
Master online yaml tools
You can pass input to this tool via
?input
query argument and it will automatically compute output. Here’s how to type it in your browser’s address bar. Click to try!
https://onlineyamltools.com/validate-yaml
?input
=plain%0Atext%0Ais%0Anot%0Ayaml
Coming soon
These yaml tools are on the way
Diff Two YAML Files
Compare two YAML files and show their differences.
Show YAML Statistics
Show detailed information about a YAML config.
Flatten YAML
Quickly flatten a nested YAML config file.
Truncate YAML
Loop over YAML and truncate lists, arrays, and strings.
Create a PHP Array from YAML
Convert YAML config to a PHP data structure.
Convert YAML to Properties
Convert a YAML file to a properties file.
Convert Properties ot YAML
Convert a properties file to a YAML file.
Convert YAML to TOML
Convert a YAML file to a TOML file.
Convert TOML to YAML
Convert a TOML config to a YAML config.
Convert YAML to NestedText
Convert a YAML file to a NestedText file.
Convert NestedText to YAML
Convert a NestedText config to a YAML config.
Convert YAML to PDF
Create a PDF file from a YAML file.
Sort YAML
Sort a YAML file by keys or by values.
Extract Keys from YAML
Find and extract all keys from a YAML file.
Extract Values from YAML
Find and extract all keys from a YAML file.
Extract Strings from YAML
Find and extract all strings from a YAML file.
Extract Numbers from YAML
Find and extract all numbers from a YAML file.
Extract Comments from YAML
Find and extract all comments from a YAML file.
Remove Comments from YAML
Clean up a YAML configuration file from comments.
Add Errors to YAML
Add tiny errors to YAML and create an invalid YAML file.
Change All Quotes to Double Quotes
Normalize all quotes in a YAML file to double quotes.
Change All Quotes to Single Quotes
Normalize all quotes in a YAML file to single quotes.
Visualize YAML Structure
Create an abstract image that represents YAML data.
YAML Validator is easy to use the YAML Validate tool. Copy, Paste, and Validate. This is also called a YAML Lint tool.
What can you do with YAML Validator Online?
- It helps to Validate your YAML data.
- It also works as YAML Checker or YAML syntax checker.
- This tool allows loading the YAML URL to validate. Use your YAML REST URL to validate. Click on the Load URL button, Enter URL and Submit.
- Users can also validate YAML Files by uploading the file. This functionality also referred to as YAML File validator
- It helps to save your validated YAML online and Share it on social sites or emails.
- YAML Validator works well on Windows, MAC, Linux, Chrome, Firefox, Edge, and Safari.
- This YAML Linter helps a developer who works with JSON data to test and verify.
Example to validate YAML
Valid YAML Try it.
InsuranceCompanies: Time: "May 2021" "Top Insurance Companies": - No: 1 Name: "Berkshire Hathaway ( BRK.A)" "Market Capitalization": "$655 billion" source: "investopedia.com" url: "https://www.investopedia.com/articles/active-trading/111314/top-10-insurance-companies-metrics.asp"
Invalid YAML Try it.
InsuranceCompanies: Time: "May 2021 "Top Insurance Companies": - No: 1 Name: "Berkshire Hathaway ( BRK.A)" "Market Capitalization": "$655 billion" source: "investopedia.com" url: "https://www.investopedia.com/articles/active-trading/111314/top-10-insurance-companies-metrics.asp"
For Advanced Users
Lua External URL
Load Lua External URL in Browser URL like this https://codebeautify.org/yaml-validator?url=external-url
https://codebeautify.org/yaml-validator?url=https://gist.githubusercontent.com/cbmgit/b4b3a9361a795e4886966b6e12abeeaf/raw/InsuranceCompanies.yaml
Text Tools
String Utilities
Case Converter
Sort Lines
Diff Checker
Text Editor
JSON Editor
Lorem Ipsum Generator
URL Parser / Query String Splitter
Slug Generator
HTML Stripper
Pastebin
Formatters
HTML Formatter / Minifier
CSS Beautifier / Minifier
JavaScript Beautifier / Minifier
JSON Formatter / Minifier
XML Formatter / Minifier
SQL Formatter / Minifier
Escape and Unescape Tools
HTML Escape / Unescape
JavaScript Escape / Unescape
JSON Escape / Unescape
XML Escape / Unescape
SQL Escape / Unescape
Validators
HTML Validator
JSON Validator
YAML Validator
Credit Card Validator
Regular Expression Tester
Cryptography and Security
Hash Generator
HMAC Generator
Checksum Calculator
Password Generator
Bcrypt Generator / Checker
UUID Generator
Encoders and Decoders
URL Encoder / Decoder
HTML Entity Encoder / Decoder
Base32 Encoder / Decoder
Base58 Encoder / Decoder
Base64 Encoder / Decoder
Base64 Image Encoder / Decoder
JWT Decoder
Converters
JavaScript / JSON Converter
CSV / JSON Converter
JSON / XML Converter
JSON / PHP Array Converter
Text / HEX Converter
Base Converter
Unit Converter
UNIX Timestamp Converter
Image Tools
Color Picker
QR Code Generator
QR Code Reader
EXIF Viewer
Network Tools
My IP
IP Lookup
IP / Number Converter
CIDR / IP Converter
WHOIS Lookup
Background
Recently, we considered an approach, where in a single file document analysts are able to share SIEM (idea) queries, and some form of documentation and or notes. We also needed to make them machine parseable and transformable, in order for us to automate the parts of the queries to feed into a SIEM system.
This sort of idea is not new or ground breaking in anyway. In fact, it is pretty popular in the information security industry to share ideas for threat detection & hunting in YAML, TOML or markdown with code blocks.
YAML and TOML file formats are used a lot in threat detection & hunting rule sharing communities, ever since Sigma — generic signature format for SIEM systems1, came out, I believe. Threat detection & hunting enthusiasts sharing ideas are also making use of YAML file format2.
In fact, vendors like Elastic3 share their detection contents on GitHub as TOML files4.
This allows some form of uniformity in how the contents should be structured and also defines how the machine or automations should extract the information.
If you have the team, time, development and engineering resources, it might be worth looking into just using Sigma and to get contents for different security systems ingesting them automatically.
However, our approach was that, we wanted something to mix SIEM specific queries and some documentations together, while only spending some time into writing a script that can just strip out the query so that it can then be fed into a SIEM system, so we went with our own YAML format.
The downsides of coming up with your own format is that, you need to first define the structure, what field and values are mandatory, what are optional and then make a decision. This downside, however, can be overcome relatively easy in some cases. This was the case for us.
NOTE: Just want the sauce?
The Details
This is how a signature format in YAML looks like:
It has a title, id, description about the rule, the author, references, logsource, detection rules.
Let’s say, for your community or organization, you decided a YAML format inspired by Sigma, however, it is not an extension, and that you do not use a single syntax high-level abstraction for your queries. It looks like:
---
id: 6068c062-627f-4d7c-9250-5059f5417726 # UUIDv4
title: some title for your detection rule
description: a short sentence about the detection rule
references:
- reference URL 1
- reference URL 2
analyst_notes: >
When you see X, you need to check if occurances of A, B, C, D are also there?
If not, it might indicate a false-positive or a scenario 1 like in Alpha.
If you see at most 3 out of 4, it is surely suspicious and therefore you should
look for to find: K, L, M, N.
query: >
SELECT * FROM registry WHERE
key LIKE 'HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\%%'
and name='Debugger';",
mitre:
- T1112
jira: PJ-1337
Now you get to a point where these files are stored in Git for version control and some form of automation is in-place, you will need to make sure that the file is properly formatted as YAML and also compliant to your custom schema.
The former can be achieved by using a linting tool like yamllint5. The latter with a library like Yamale6, which is what we went with.
YAML Linting
yamllint is a command line tool and a library you can use in your own tooling. If we run the tool on the YAML file above:
$ yamllint detection_rule.yml
detection_rule.yml
2:42 warning too few spaces before comment (comments)
11:81 error line too long (81 > 80 characters) (line-length)
15:81 error line too long (102 > 80 characters) (line-length)
However, if you would like to combine such a linting process together with other checks your scripts are doing, you can import the library7. Here is an example:
import yamllint
from yamllint.config import YamlLintConfig
raw_yaml = open('detection_rule.yml', 'r').read()
yaml_config = YamlLintConfig("extends: default")
for p in yamllint.linter.run(raw_yaml, yaml_config):
print(p.desc, p.line, p.rule)
Yamale — schema validation
Yamale is also a command line tool and a library. It comes with a few default validator8 functions, and is also very easily extendable. Here we will see how we could extend it for our schema.
First, we need to come up with a schema dictionary that Yamale can understand to use to validator your YAML files. Let’s consider the following as an initial schema dictionary:
id: str()
title: str()
description: str()
references: list(str())
analyst_notes: str()
query: str()
mitre: list(str())
jira: str()
Running yamale and providing the above schema as follows, yields a validation success.
$ yamale -s schema.yaml detection_rule.yml
Validating /home/user/project-x/detection_rule.yml...
Validation success! 👍
If you kept a close eye, you will have noticed that we initially told yamale that id is a string, however, that is not entirely true.
The validation will also pass if you wrote in a bogus string that is not a UUID. So we will need to extend yamale and write our own validator.
Looking at an example custom validator9 in their example, we can try a proof of concept UUID validator and also include the validation routines as well:
import yamale
import uuid
from yamale.validators import DefaultValidators, Validator
class UUID(Validator):
""" Custom UUID validator """
tag = 'uuid'
def _is_valid(self, value):
try:
luuid = uuid.UUID(str(value))
except ValueError:
return False
return True
validators = DefaultValidators.copy() # This is a dictionary
validators[UUID.tag] = UUID
schema = yamale.make_schema('./schema.yaml', validators=validators)
data = yamale.make_data('./detection_rule.yml')
try:
yamale.validate(schema, data)
print('Validation success! 👍')
except ValueError as e:
print('Validation failed!n%s' % str(e))
exit(1)
Edit the validator for id in schema.yaml to uuid(), run the script above and it should output:
$ python schema-validate.py
Validation success! 👍
Say, for another custom validator, you want to check and make sure that jira values do confirm to the documented JIRA project key format10, since these are manually entered by analysts. Here we will reuse/sub-class from the built-in Regex validator.
import re
from yamale.validators import Regex
class JIRA(Regex):
""" Custom JIRA Project ID validator. """
tag = 'jira'
def __init__(self, *args, **kwargs):
self._project_key = str(kwargs.pop('project_key', ''))
super(JIRA, self).__init__(*args, **kwargs)
if len(self._project_key) > 0:
self.regexes = [
re.compile("^%s-d+$" % (self._project_key))
]
else:
self.regexes = [
re.compile("^([A-Z]{2}[0-9]{2})-d+$"),
re.compile("^([A-Z][A-Z_0-9]+)-d+$"),
]
Adjust jira field’s value in schema.yaml as jira(project_key='JP') and on running the script, it should error out:
Validation failed!
Error validating data './detection_rule.yml' with schema './schema.yaml'
jira: 'PJ-1337' is not a jira match.
Challenge
Let us try writing a schema validator for the Sigma rule we mentioned at the beginning.
---
title: str(min=1, max=256)
id: str()
status: enum('stable', 'testing', 'experimental', required=False)
description: str(required=False)
author: str(required=False)
date: str()
modified: str()
references: list(str(), required=False)
tags: list(str())
logsource: include('logsource')
detection: include('detection')
falsepositives: any(str(), list(), required=False)
level: enum('low', 'medium', 'high', 'critical', required=False)
---
logsource:
product: str(required=False)
category: str(required=False)
service: str(required=False)
definition: str(required=False)
---
detection:
selection: any(str(), list(), map(key=str()))
condition: str()
timeframe: str(required=False)
Note: At times if you compare Yamale with something like Rx11, the former seems somewhat limiting for the way Sigma was designed. However, I would start with something like Yamale first and then think about Rx later on.
Full Example Files
detection_rule.yml
---
id: 6068c062-627f-4d7c-9250-5059f5417726
title: some title for your detection rule
description: a short sentence about the detection rule
references:
- reference URL 1
- reference URL 2
analyst_notes: >
When you see X, you need to check if occurances of A, B, C, D are also there?
If not, it might indicate a false-positive or a scenario 1 like in Alpha.
If you see at most 3 out of 4, it is surely suspicious and therefore you should
look for to find: K, L, M, N.
query:
SELECT * FROM registry WHERE
key LIKE 'HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\%%'
and name='Debugger';",
mitre:
- T1112
jira: PJ-1337
schema.yaml
id: uuid()
title: str()
description: str()
references: list(str())
analyst_notes: str()
query: str()
mitre: list(str())
jira: jira()
validator.py
import yamale
import uuid
import re
from yamale.validators import DefaultValidators, Validator, Regex
class UUID(Validator):
""" Custom UUID validator """
tag = 'uuid'
def __init__(self, *args, **kwargs):
super(UUID, self).__init__(*args, **kwargs)
self._version = int(kwargs.pop('version', 4))
def _is_valid(self, value):
try:
luuid = uuid.UUID(str(value), version=self._version)
except ValueError:
return False
return True
class JIRA(Regex):
""" Custom JIRA Project ID validator. """
tag = 'jira'
def __init__(self, *args, **kwargs):
self._project_key = str(kwargs.pop('project_key', ''))
super(JIRA, self).__init__(*args, **kwargs)
if len(self._project_key) > 0:
self.regexes = [
re.compile("^%s-d+$" % (self._project_key))
]
else:
self.regexes = [
re.compile("^([A-Z]{2}[0-9]{2})-d+$"),
re.compile("^([A-Z][A-Z_0-9]+)-d+$"),
]
validators = DefaultValidators.copy() # This is a dictionary
validators[UUID.tag] = UUID
validators[JIRA.tag] = JIRA
schema = yamale.make_schema('./schema.yaml', validators=validators)
data = yamale.make_data('./detection_rule.yml')
try:
yamale.validate(schema, data)
print('Validation success! 👍')
except ValueError as e:
print('Validation failed!n%s' % str(e))
exit(1)

