2 posts tagged with "storage"

ApudFlow now supports MongoDB for document storage with built-in user isolation. Whether you're building user profiles, storing analytics data, or managing content, MongoDB workers provide flexible document storage with automatic security.

What is MongoDB?​

MongoDB is a document database that stores data in flexible, JSON-like documents. Unlike traditional relational databases, MongoDB allows you to store complex, nested data structures without predefined schemas.

Key Features​

Automatic User Isolation​

Every document is automatically tagged with your user ID, ensuring complete data separation between users. No cross-tenant data leaks or security concerns.

Two Connection Modes​

• Internal Mode: Managed connection with a single built-in collection. No setup required - start storing data instantly.
• External Mode: Connect to your own MongoDB database and collections for full control.

Rich Query Capabilities​

Perform complex queries using MongoDB's powerful query language. Filter by any field, use regex patterns, and aggregate data across documents.

What Can You Use It For?​

User Data Management​

Store user profiles, preferences, and settings. Perfect for applications that need to maintain user-specific data across sessions.

Example: Store user trading preferences
- operation: insert
- document: {"theme": "dark", "defaultTimeframe": "1H", "alertsEnabled": true}

Example: User notification settings
- operation: insert
- document: {
  "emailNotifications": true,
  "smsAlerts": false,
  "notificationTypes": ["price_alerts", "news", "portfolio_updates"],
  "quietHours": {"start": "22:00", "end": "08:00"}
}

Analytics and Metrics​

Collect and analyze usage data, performance metrics, or business intelligence data. MongoDB's flexible schema makes it easy to evolve your data structure as needs change.

Example: Track user interactions
- operation: insert
- document: {"action": "chart_view", "symbol": "AAPL", "timestamp": "2025-10-21T10:00:00Z"}

Example: Performance analytics
- operation: insert
- document: {
  "workflowId": "market_scanner",
  "executionTime": 2.3,
  "dataPoints": 15420,
  "success": true,
  "timestamp": "2025-10-21T10:00:00Z"
}

Example: A/B test results
- operation: insert
- document: {
  "testId": "new_ui_layout",
  "variant": "A",
  "userId": "user123",
  "conversion": true,
  "timeSpent": 45
}

Content Management​

Store articles, blog posts, comments, or any structured content. The document model is perfect for content with varying fields and nested data.

Example: Save analysis reports
- operation: insert
- document: {
  "title": "Q4 Earnings Analysis",
  "content": "...",
  "tags": ["earnings", "analysis"],
  "metadata": {"wordCount": 1250, "readTime": 6}
}

Example: Product catalog
- operation: insert
- document: {
  "name": "Premium Analytics Package",
  "description": "Advanced market analysis tools",
  "pricing": {"monthly": 49.99, "yearly": 499.99},
  "features": ["real-time data", "custom indicators", "alerts"],
  "category": "analytics"
}

Workflow State Persistence​

Save intermediate workflow results, cache expensive computations, or maintain state between workflow runs.

Example: Cache API responses
- operation: insert
- document: {"endpoint": "/api/market-data", "response": {...}, "cachedAt": "2025-10-21T10:00:00Z"}

Example: Workflow checkpoints
- operation: insert
- document: {
  "workflowId": "data_import_001",
  "step": "data_validation",
  "progress": 75,
  "lastProcessedId": 15420,
  "errors": []
}

E-commerce Applications​

Manage product inventories, customer orders, and shopping cart data with complex nested structures.

Example: Order management
- operation: insert
- document: {
  "orderId": "ORD-2025-001",
  "customerId": "user123",
  "items": [
    {"productId": "AAPL", "quantity": 100, "price": 150.25},
    {"productId": "GOOGL", "quantity": 50, "price": 2800.00}
  ],
  "total": 140125.00,
  "status": "pending"
}

IoT and Sensor Data​

Store time-series data from sensors, devices, or monitoring systems with flexible schemas.

Example: Sensor readings
- operation: insert
- document: {
  "deviceId": "sensor_001",
  "timestamp": "2025-10-21T10:00:00Z",
  "readings": {
    "temperature": 23.5,
    "humidity": 65.2,
    "pressure": 1013.25
  },
  "location": {"lat": 40.7128, "lng": -74.0060}
}

Gaming and User Progress​

Track player statistics, achievements, inventory, and game state with nested data structures.

Example: Player profile
- operation: insert
- document: {
  "playerId": "player123",
  "level": 25,
  "experience": 15420,
  "inventory": [
    {"item": "golden_pickaxe", "quantity": 1},
    {"item": "diamond_ore", "quantity": 50}
  ],
  "achievements": ["first_trade", "profit_master", "speed_trader"]
}

Financial Markets & Trading​

Store complex financial data, trading strategies, market analysis, and portfolio management information with flexible document structures.

Example: Trading strategy configuration
- operation: insert
- document: {
  "strategyName": "Momentum Crossover",
  "parameters": {
    "fastPeriod": 9,
    "slowPeriod": 21,
    "stopLoss": 0.02,
    "takeProfit": 0.05
  },
  "symbols": ["AAPL", "MSFT", "GOOGL"],
  "active": true,
  "performance": {"winRate": 0.68, "avgReturn": 0.034}
}

Example: Portfolio holdings
- operation: insert
- document: {
  "portfolioId": "growth_portfolio",
  "holdings": [
    {"symbol": "AAPL", "shares": 150, "avgPrice": 145.20, "currentValue": 22530.00},
    {"symbol": "MSFT", "shares": 75, "avgPrice": 280.50, "currentValue": 21037.50}
  ],
  "totalValue": 43567.50,
  "lastUpdated": "2025-10-21T10:00:00Z"
}

Example: Market analysis reports
- operation: insert
- document: {
  "symbol": "AAPL",
  "analysisType": "technical",
  "indicators": {
    "rsi": 65.4,
    "macd": {"signal": 2.15, "histogram": 0.85},
    "movingAverages": {"sma20": 152.30, "sma50": 148.75}
  },
  "recommendation": "BUY",
  "confidence": 0.82,
  "generatedAt": "2025-10-21T10:00:00Z"
}

Example: Trade execution log
- operation: insert
- document: {
  "tradeId": "T20251021001",
  "symbol": "AAPL",
  "side": "BUY",
  "quantity": 100,
  "price": 150.25,
  "timestamp": "2025-10-21T10:00:00Z",
  "strategy": "momentum_crossover",
  "fees": 1.50,
  "executionDetails": {
    "venue": "NASDAQ",
    "orderType": "MARKET",
    "slippage": 0.05
  }
}

Example: Risk management data
- operation: insert
- document: {
  "riskProfile": "moderate",
  "positionLimits": {
    "maxSinglePosition": 0.1,
    "maxSectorExposure": 0.25,
    "maxDrawdown": 0.15
  },
  "varCalculation": {
    "confidence": 0.95,
    "timeHorizon": 1,
    "value": 12500.00
  },
  "lastAssessment": "2025-10-21T10:00:00Z"
}

Example: Economic indicators database
- operation: insert
- document: {
  "indicator": "GDP_Growth",
  "country": "US",
  "frequency": "quarterly",
  "values": [
    {"date": "2025-Q2", "value": 2.1, "forecast": 2.3},
    {"date": "2025-Q3", "value": null, "forecast": 2.0}
  ],
  "source": "Bureau of Economic Analysis",
  "lastUpdated": "2025-10-21T10:00:00Z"
}

Getting Started​

1. Add a "MongoDB" worker to your flow
2. Choose connection mode (internal for quick start, external for custom setup)
3. Select operation (insert, find, update, delete, count)
4. Configure your document or query parameters

Example Workflow: User Portfolio Tracker​

Create a workflow that:

1. Fetches user portfolio data from an API
2. Stores it in MongoDB with user isolation
3. Updates analytics counters
4. Sends notifications based on stored preferences

This combination of document storage with workflow automation opens up endless possibilities for data-driven applications.

Performance & Scalability​

MongoDB excels at handling large volumes of data with fast read/write operations. The document model scales naturally as your data structure evolves, making it perfect for growing applications.

Start building with MongoDB today and unlock the power of flexible document storage in your workflows!

ApudFlow now includes Redis support for high-performance key-value storage. Perfect for caching, session management, and real-time data operations that require speed and reliability.

What is Redis?​

Redis is an in-memory data structure store used as a database, cache, and message broker. It provides sub-millisecond response times and supports various data structures like strings, hashes, lists, sets, and more.

Key Features​

Lightning Fast Performance​

Redis stores data in memory, delivering microsecond response times. Perfect for applications requiring instant data access.

Rich Data Structures​

Beyond simple key-value pairs, Redis supports:

• Strings (text, numbers, binary data)
• Hashes (field-value objects)
• Lists (ordered collections)
• Sets (unique value collections)
• Sorted sets (ranked collections)

Automatic Expiration​

Set TTL (time-to-live) on keys for automatic cleanup. Perfect for temporary data like sessions, cache entries, and rate limiting.

User Isolation​

All keys are automatically prefixed with your user ID, ensuring complete data separation and security.

What Can You Use It For?​

Caching API Responses​

Speed up your workflows by caching expensive API calls, database queries, or computational results.

Example: Cache market data
- operation: set
- key: "market_data_AAPL"
- value: {"price": 150.25, "volume": 4523000}
- ttl: 300 (5 minutes)

Session Management​

Store user session data, preferences, and temporary state with automatic expiration.

Example: User session storage
- operation: set
- key: "session_12345"
- value: {"userId": 12345, "lastActivity": "2025-10-21T10:00:00Z"}
- ttl: 3600 (1 hour)

Rate Limiting​

Implement rate limiting for API calls, user actions, or workflow executions.

Example: API rate limiting
- operation: incr
- key: "api_calls_user_12345"
- value: 1
- Then check if value exceeds your limit

Real-Time Analytics​

Track counters, metrics, and statistics with atomic operations.

Example: Page view tracking
- operation: incr
- key: "page_views_dashboard"
- value: 1

Leaderboards and Rankings​

Use sorted sets for gaming leaderboards, popularity rankings, or priority queues.

Example: Trading volume leaderboard
- operation: set (with sorted set operations)
- key: "trading_volume"
- Add scores for different symbols

Message Queues​

Implement simple message queues for background processing or event handling.

Example: Notification queue
- operation: set (using lists)
- key: "notification_queue"
- Push messages for later processing

Getting Started​

1. Add a "Redis" worker to your flow
2. Choose operation (set, get, delete, list, exists, incr, expire)
3. Configure key, value, and optional parameters like TTL

Example Workflow: Smart Caching System​

Create a workflow that:

1. Checks Redis cache for data
2. If cache miss, fetches from external API
3. Stores result in Redis with TTL
4. Returns cached or fresh data

This pattern dramatically improves performance and reduces API costs.

Performance Benefits​

Redis typically delivers:

• 100x faster than traditional databases for simple operations
• Sub-millisecond latency for most operations
• High throughput (100k+ operations per second)
• Atomic operations ensuring data consistency

Use Cases by Industry​

Finance & Trading​

• Real-time price caching
• Portfolio position storage
• Rate limiting for trading APIs
• Session management for trading platforms
• Order book management
• Trade execution tracking
• Risk limit monitoring
• Market data snapshots
• Alert system state
• Trading strategy parameters
• Historical price buffers
• Account balance caching
• Margin requirement tracking
• Stop-loss order management
• Arbitrage opportunity detection

Algorithmic Trading Excellence
Redis excels in algorithmic trading due to its lightning-fast access speeds and atomic operations. High-frequency trading strategies require microsecond response times for market data, order execution, and risk management. Redis enables real-time strategy execution, instant position updates, and rapid arbitrage detection across multiple exchanges. Its pub/sub capabilities allow for real-time signal broadcasting to multiple trading algorithms, while sorted sets efficiently manage order books and priority queues for trade execution.

In algorithmic trading, every millisecond counts. Redis's in-memory storage eliminates disk I/O bottlenecks, ensuring strategies can react instantly to market movements. Complex algorithms can maintain state across multiple timeframes, track position sizes with atomic increments, and implement sophisticated risk controls. The ability to expire keys automatically handles time-sensitive data like quotes and orders, while Redis clustering provides the scalability needed for high-volume trading operations. Whether you're running statistical arbitrage, market making, or momentum strategies, Redis provides the performance foundation that separates profitable algorithms from those that lag behind.

E-commerce​

• Shopping cart storage
• Product catalog caching
• User session management
• Inventory counters

Gaming​

• Player statistics
• Leaderboards
• Session storage
• Achievement tracking

Analytics​

• Event counting
• User behavior tracking
• A/B test results
• Real-time dashboards

Redis brings enterprise-grade performance to your workflows. Start using it today for faster, more reliable data operations!