Skip to main content

Data Sources

Arxignis leverages a comprehensive collection of over 30 threat intelligence sources to provide accurate and up-to-date security scoring. Our data collection strategy combines multiple source types to ensure comprehensive coverage and high-quality threat intelligence.

Source Categories​

Community Sources​

Community-driven threat intelligence feeds that are freely available and maintained by security researchers and organizations:

  • IP reputation databases and blocklists
  • Open source threat intelligence platforms
  • Community-maintained malware and botnet intelligence
  • Brute force and attack pattern detection feeds
  • Spam and malicious activity tracking
  • Network scanning and reconnaissance data
  • Proxy and anonymization service detection
  • Tor network infrastructure tracking
  • Banking trojan and financial threat intelligence
  • High-confidence threat intelligence feeds

Free Sources​

Open source and freely available threat intelligence feeds:

  • Commercial-grade threat intelligence with free tiers
  • Regional security community blocklists
  • Community intelligence scoring systems
  • Open source security research data

Premium threat intelligence sources that require authentication and provide high-quality, curated data:

  • Advanced threat detection and response platforms
  • Proprietary threat intelligence feeds
  • Commercial security vendor data
  • Enterprise-grade threat intelligence services

Data Collection Architecture​

Source Management​

Each data source is configured with:

  • Source Metadata: Name, URL, threat level, type, and expiration settings
  • Parser Configuration: Specific parsing logic for different data formats
  • Authentication: Support for both public and authenticated sources
  • Caching Strategy: Intelligent caching to optimize performance and reduce API calls

Parser Types​

The system supports multiple parsing formats to handle diverse data sources:

  • TXT Parser: Simple text format with one indicator per line
  • CSV Parser: Comma-separated value format
  • JSON Parser: Structured JSON data format
  • CrowdSec Parser: Specialized parser for CrowdSec data format
  • DangerRulez Parser: Custom parser for DangerRulez format
  • Flexible Parser: Adaptive parser that can handle various formats

Data Processing Pipeline​

  1. Collection: Automated data fetching from all configured sources
  2. Parsing: Format-specific parsing to extract threat indicators
  3. Validation: IP address validation and deduplication
  4. Scoring: Threat level assignment based on source reputation
  5. Tagging: Automatic categorization using threat intelligence tags
  6. Storage: Efficient storage in PostgreSQL with relationship mapping

Threat Intelligence Tags​

The system categorizes threats using a comprehensive tagging system:

  • Malware (Weight: 9) - Malicious software and code
  • Botnet (Weight: 8) - Botnet command and control infrastructure
  • C2 (Weight: 9) - Command and control servers
  • Phishing (Weight: 7) - Phishing and social engineering attacks
  • Proxy (Weight: 4) - Proxy and anonymization services
  • Tor (Weight: 5) - Tor network infrastructure
  • Scanner (Weight: 3) - Network scanning and reconnaissance
  • VPN (Weight: 2) - VPN services and infrastructure
  • Spam (Weight: 2) - Spam and unwanted communications
  • Brute Force (Weight: 6) - Brute force attack patterns
  • Unknown (Weight: 1) - Unclassified or unknown threats

Data Quality and Reliability​

Source Validation​

  • Threat Level Scoring: Each source is assigned a threat level (1-15) based on reliability and accuracy
  • Expiration Management: Automatic data refresh based on source-specific expiration times
  • Error Handling: Robust error handling and retry mechanisms for failed data collection
  • Status Monitoring: Real-time monitoring of source collection status

Data Freshness​

  • Real-time Updates: Continuous data collection and processing
  • Cache Management: Intelligent caching to balance freshness and performance
  • Source Health: Monitoring and alerting for source availability and data quality

Integration and Scalability​

API Integration​

  • RESTful APIs: Standardized API endpoints for data access
  • Authentication: Secure API access with authentication requirements
  • Rate Limiting: Intelligent rate limiting to respect source API limits

Performance Optimization​

  • Parallel Processing: Concurrent data collection from multiple sources
  • Caching Strategy: Multi-level caching for optimal performance
  • Database Optimization: Efficient database design with proper indexing

Proprietary Data​

Arxignis also incorporates proprietary threat intelligence data:

  • Internal Threat Data: Data collected from Arxignis security infrastructure
  • Custom Indicators: Proprietary threat indicators and patterns
  • Behavioral Analysis: Advanced behavioral threat detection
  • Machine Learning Models: AI-powered threat classification and scoring

This comprehensive approach to data collection ensures that Arxignis provides the most accurate and up-to-date threat intelligence available, combining the best of community, commercial, and proprietary sources.

Score​

You can find more information about our scoring system