Multi-Market Implementation: Phase 4 - Cache Invalidation and Repository Updates

Status: Planning

Dependencies

Requires completed: - Phase 1 (Entities) - Multi-market entities exist - Phase 2 (Migration) - Fields added to existing entities - Phase 3 (Services) - BuyingOptionService uses cache tags

Blocks: - Phase 5 (API Integration) - API uses repository methods - All subsequent phases

📋 Todo Checklist

• [ ] Extend CacheInvalidationSubscriber for multi-market entities
• [ ] Add availability methods to CoffeeBeanRepository
• [ ] Update CoffeeBeanRepository::findByRequest() for market-based filtering
• [ ] Add cache tag constants to entities
• [ ] Test cache invalidation for all entity changes
• [ ] Test repository methods with market joins
• [ ] Verify N+1 query prevention
• [ ] Document cache tag naming conventions

🔍 Analysis & Investigation

Problem Statement

The current cache invalidation system needs to handle multi-market entities: - Market changes should invalidate buying URLs and bean lists - AffiliateProgram changes should invalidate buying URLs - BuyingOption changes should invalidate specific bean URLs - RoasterCrawlConfig and CrawlUrl changes should invalidate related caches

Additionally, repository methods need to support market-based filtering for availability.

Current Architecture

CacheInvalidationSubscriber: - Listens to Doctrine entity changes - Tags caches based on entity type and ID - Currently handles: CoffeeBean, Roaster, Country, etc.

CoffeeBeanRepository: - Has findByRequest() for filtering beans - Currently uses CoffeeBean.available field (to be replaced) - No market-aware availability filtering yet

Target Architecture

CacheInvalidationSubscriber: - Extended with multi-market entity handlers - Comprehensive tag mapping for cache invalidation - Handles cascading invalidations (e.g., Market → all buying URLs)

CoffeeBeanRepository: - New methods: isAvailableInAnyMarket(), isAvailableForCountry() - Updated findByRequest() with market-based availability filtering - Optimized joins to prevent N+1 queries

📝 Implementation Plan

Prerequisites

• Phase 1, 2, 3 completed
• Tag-aware cache configured (api.cache)
• Doctrine event listeners configured

Step-by-Step Implementation

Step 1: Extend CacheInvalidationSubscriber

File: src/EventListener/CacheInvalidationSubscriber.php (modify existing)

Add multi-market entity handlers to the tag mapping:

<?php

namespace App\EventListener;

use App\Entity\AffiliateProgram;
use App\Entity\BuyingOption;
use App\Entity\CoffeeBean;
use App\Entity\CrawlUrl;
use App\Entity\Market;
use App\Entity\RoasterCrawlConfig;
use Doctrine\Bundle\DoctrineBundle\Attribute\AsDoctrineListener;
use Doctrine\ORM\Event\OnFlushEventArgs;
use Doctrine\ORM\Events;
use Symfony\Contracts\Cache\TagAwareCacheInterface;

#[AsDoctrineListener(event: Events::onFlush)]
final readonly class CacheInvalidationSubscriber
{
    public function __construct(
        private TagAwareCacheInterface $apiCache
    ) {}

    public function onFlush(OnFlushEventArgs $args): void
    {
        $em = $args->getObjectManager();
        $uow = $em->getUnitOfWork();

        $tagsToInvalidate = [];

        // Collect tags from all changed entities
        foreach ($uow->getScheduledEntityInsertions() as $entity) {
            $tagsToInvalidate = array_merge($tagsToInvalidate, $this->getTagsForEntity($entity));
        }

        foreach ($uow->getScheduledEntityUpdates() as $entity) {
            $tagsToInvalidate = array_merge($tagsToInvalidate, $this->getTagsForEntity($entity));
        }

        foreach ($uow->getScheduledEntityDeletions() as $entity) {
            $tagsToInvalidate = array_merge($tagsToInvalidate, $this->getTagsForEntity($entity));
        }

        // Invalidate all collected tags
        if (!empty($tagsToInvalidate)) {
            $this->apiCache->invalidateTags(array_unique($tagsToInvalidate));
        }
    }

    /**
     * Get cache tags to invalidate for a specific entity.
     */
    private function getTagsForEntity(object $entity): array
    {
        return match (true) {
            $entity instanceof Market => [
                'market_' . $entity->getId(),
                'buying_urls',
                'coffee_beans_list',
                'coffee_beans_detail',
            ],

            $entity instanceof AffiliateProgram => [
                'affiliate_program_' . $entity->getId(),
                'buying_urls',
                'coffee_beans_list',
                'coffee_beans_detail',
            ],

            $entity instanceof BuyingOption => [
                'buying_option_' . $entity->getCoffeeBean()->getId() . '_' . $entity->getMarket()->getId(),
                'coffee_bean_' . $entity->getCoffeeBean()->getId(),
                'market_' . $entity->getMarket()->getId(),
                'buying_urls',
                'coffee_beans_detail',
            ],

            $entity instanceof RoasterCrawlConfig => [
                'roaster_config_' . $entity->getId(),
                'buying_urls',
                'coffee_beans_list',
                'coffee_beans_detail',
            ],

            $entity instanceof CrawlUrl => [
                'coffee_bean_' . $entity->getCoffeeBean()->getId(),
                'buying_urls',
                'coffee_beans_detail',
            ],

            $entity instanceof CoffeeBean => [
                'coffee_bean_' . $entity->getId(),
                'coffee_beans_list',
                'coffee_beans_detail',
            ],

            // ... existing entity handlers (Roaster, Country, etc.)

            default => [],
        };
    }
}

Cache Tag Naming Conventions:

Tag Pattern	Purpose	Example
market_{id}	Specific market changes	market_550e8400-e29b-41d4-a716-446655440000
affiliate_program_{id}	Specific affiliate program changes	affiliate_program_123
buying_option_{bean}_{market}	Specific buying option	buying_option_bean123_market456
roaster_config_{id}	Specific config changes	roaster_config_789
coffee_bean_{id}	Specific bean changes	coffee_bean_abc123
buying_urls	All buying URL resolutions	buying_urls
coffee_beans_list	Bean listing endpoints	coffee_beans_list
coffee_beans_detail	Bean detail endpoints	coffee_beans_detail

Step 2: Add Availability Methods to CoffeeBeanRepository

File: src/Repository/CoffeeBeanRepository.php (modify existing)

Add methods for market-aware availability checking:

<?php

namespace App\Repository;

use App\Entity\CoffeeBean;
use Doctrine\Bundle\DoctrineBundle\Repository\ServiceEntityRepository;
use Doctrine\Persistence\ManagerRegistry;

class CoffeeBeanRepository extends ServiceEntityRepository
{
    public function __construct(ManagerRegistry $registry)
    {
        parent::__construct($registry, CoffeeBean::class);
    }

    /**
     * Check if a coffee bean is available in any market.
     *
     * @param CoffeeBean $bean Coffee bean to check
     * @return bool True if bean has at least one available CrawlUrl
     */
    public function isAvailableInAnyMarket(CoffeeBean $bean): bool
    {
        $qb = $this->createQueryBuilder('cb')
            ->select('COUNT(cu.id)')
            ->join('cb.crawlUrls', 'cu')
            ->where('cb.id = :beanId')
            ->andWhere('cu.available = true')
            ->setParameter('beanId', $bean->getId());

        return $qb->getQuery()->getSingleScalarResult() > 0;
    }

    /**
     * Check if a coffee bean is available for a specific country.
     *
     * Checks if bean has any CrawlUrl that:
     * - Is marked as available
     * - From an active RoasterCrawlConfig
     * - With an active Market serving the specified country
     *
     * @param CoffeeBean $bean Coffee bean to check
     * @param string $visitorCountryId Country ID (2-letter code)
     * @return bool True if bean is available for the specified country
     */
    public function isAvailableForCountry(CoffeeBean $bean, string $visitorCountryId): bool
    {
        $qb = $this->createQueryBuilder('cb')
            ->select('COUNT(cu.id)')
            ->join('cb.crawlUrls', 'cu')
            ->join('cu.roasterCrawlConfig', 'rcc')
            ->join('rcc.market', 'm')
            ->join('m.countries', 'c')
            ->where('cb.id = :beanId')
            ->andWhere('c.id = :countryId')
            ->andWhere('cu.available = true')
            ->andWhere('m.isActive = true')
            ->andWhere('rcc.active = true')
            ->setParameter('beanId', $bean->getId())
            ->setParameter('countryId', $visitorCountryId);

        return $qb->getQuery()->getSingleScalarResult() > 0;
    }

    // ... existing methods ...
}

Step 3: Update findByRequest() for Market-Based Filtering

File: src/Repository/CoffeeBeanRepository.php (modify existing method)

Update the availability filtering logic:

/**
 * Find coffee beans by API request parameters.
 *
 * @param array $filters Request filters (available, visitorCountryId, etc.)
 * @return CoffeeBean[]
 */
public function findByRequest(array $filters): array
{
    $queryBuilder = $this->createQueryBuilder('cb');

    // ... existing filters (roaster, origin, processing, etc.) ...

    // UPDATED: Market-aware availability filtering
    if (isset($filters['available'])) {
        $available = (bool) $filters['available'];
        $visitorCountryId = $filters['visitorCountryId'] ?? null;

        if ($visitorCountryId !== null) {
            // Filter by CrawlUrls from configs serving visitor country
            $queryBuilder
                ->join('cb.crawlUrls', 'cu_avail')
                ->join('cu_avail.roasterCrawlConfig', 'rcc_avail')
                ->join('rcc_avail.market', 'm_avail')
                ->join('m_avail.countries', 'c_avail')
                ->andWhere('c_avail.id = :visitorCountryId')
                ->andWhere('cu_avail.available = :available')
                ->andWhere('m_avail.isActive = true')
                ->andWhere('rcc_avail.active = true')
                ->setParameter('visitorCountryId', $visitorCountryId)
                ->setParameter('available', $available);
        } else {
            // Filter by ANY CrawlUrl availability
            $queryBuilder
                ->join('cb.crawlUrls', 'cu_avail')
                ->andWhere('cu_avail.available = :available')
                ->setParameter('available', $available);
        }

        // IMPORTANT: Use DISTINCT to prevent duplicate results from multiple CrawlUrls
        $queryBuilder->distinct();
    }

    // ... rest of method (sorting, pagination, etc.) ...

    return $queryBuilder->getQuery()->getResult();
}

Key changes: - Replace cb.available with cu_avail.available - Add joins to Market and Country when visitorCountryId is provided - Use DISTINCT to prevent duplicate beans from multiple CrawlUrls - Filter by active markets and configs

Step 4: Optimize Repository Queries with Joins

File: src/Repository/CoffeeBeanRepository.php

Add method for loading beans with all needed relationships:

/**
 * Find coffee beans with all relationships loaded (prevents N+1 queries).
 *
 * Use this method when you need to access CrawlUrls, Markets, etc.
 *
 * @param array $beanIds Array of bean IDs to load
 * @return CoffeeBean[]
 */
public function findWithRelationships(array $beanIds): array
{
    return $this->createQueryBuilder('cb')
        ->select('cb', 'cu', 'rcc', 'm', 'ap', 'shipCountries')
        ->leftJoin('cb.crawlUrls', 'cu')
        ->leftJoin('cu.roasterCrawlConfig', 'rcc')
        ->leftJoin('rcc.market', 'm')
        ->leftJoin('m.affiliateProgram', 'ap')
        ->leftJoin('rcc.shipsTo', 'shipCountries')
        ->where('cb.id IN (:beanIds)')
        ->setParameter('beanIds', $beanIds)
        ->getQuery()
        ->getResult();
}

Usage: Call this method before passing beans to BuyingOptionService::getUrlForVisitor() to prevent N+1 queries.

Step 5: Add Cache Tag Constants

File: src/Service/Cache/CacheTags.php (create new)

Define constants for cache tags:

<?php

namespace App\Service\Cache;

final class CacheTags
{
    // Entity-specific tags
    public const MARKET = 'market_%s';
    public const AFFILIATE_PROGRAM = 'affiliate_program_%s';
    public const BUYING_OPTION = 'buying_option_%s_%s'; // bean_id, market_id
    public const ROASTER_CONFIG = 'roaster_config_%s';
    public const COFFEE_BEAN = 'coffee_bean_%s';
    public const COUNTRY = 'country_%s';

    // Global tags
    public const BUYING_URLS = 'buying_urls';
    public const COFFEE_BEANS_LIST = 'coffee_beans_list';
    public const COFFEE_BEANS_DETAIL = 'coffee_beans_detail';

    /**
     * Generate tag for a specific market.
     */
    public static function market(string $marketId): string
    {
        return sprintf(self::MARKET, $marketId);
    }

    /**
     * Generate tag for a specific affiliate program.
     */
    public static function affiliateProgram(string $programId): string
    {
        return sprintf(self::AFFILIATE_PROGRAM, $programId);
    }

    /**
     * Generate tag for a specific buying option.
     */
    public static function buyingOption(string $beanId, string $marketId): string
    {
        return sprintf(self::BUYING_OPTION, $beanId, $marketId);
    }

    /**
     * Generate tag for a specific roaster config.
     */
    public static function roasterConfig(string $configId): string
    {
        return sprintf(self::ROASTER_CONFIG, $configId);
    }

    /**
     * Generate tag for a specific coffee bean.
     */
    public static function coffeeBean(string $beanId): string
    {
        return sprintf(self::COFFEE_BEAN, $beanId);
    }
}

Update CacheInvalidationSubscriber to use constants.

Step 6: Create Cache Invalidation Tests

File: tests/EventListener/CacheInvalidationSubscriberTest.php

<?php

namespace App\Tests\EventListener;

use App\Entity\Market;
use App\Entity\AffiliateProgram;
use App\Entity\BuyingOption;
use App\Entity\CoffeeBean;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;
use Symfony\Contracts\Cache\TagAwareCacheInterface;

class CacheInvalidationSubscriberTest extends KernelTestCase
{
    private TagAwareCacheInterface $cache;

    protected function setUp(): void
    {
        self::bootKernel();
        $this->cache = self::getContainer()->get('api.cache');
    }

    public function testMarketChangeInvalidatesCache(): void
    {
        $em = self::getContainer()->get('doctrine')->getManager();

        // Create and cache a value with market tag
        $cacheKey = 'test_market_cache';
        $this->cache->get($cacheKey, function ($item) {
            $item->tag(['market_test-id', 'buying_urls']);
            return 'test value';
        });

        // Verify cache exists
        $this->assertEquals('test value', $this->cache->get($cacheKey, fn() => 'fallback'));

        // Create and persist a market (triggers cache invalidation)
        $market = new Market();
        $market->setName('Test Market');
        // ... set required fields ...

        $em->persist($market);
        $em->flush();

        // Verify cache was invalidated
        $result = $this->cache->get($cacheKey, fn() => 'cache was invalidated');
        $this->assertEquals('cache was invalidated', $result);
    }

    public function testBuyingOptionChangeInvalidatesSpecificBean(): void
    {
        // Test that changing a BuyingOption invalidates:
        // - buying_option_{bean}_{market} tag
        // - coffee_bean_{id} tag
        // - market_{id} tag
        // - buying_urls tag
    }

    // ... more test cases ...
}

Step 7: Create Repository Tests

File: tests/Repository/CoffeeBeanRepositoryTest.php (extend existing)

Add tests for new methods:

public function testIsAvailableInAnyMarket(): void
{
    // Create bean with one available and one unavailable CrawlUrl
    $bean = $this->createBeanWithCrawlUrls([
        ['available' => true],
        ['available' => false],
    ]);

    $result = $this->repository->isAvailableInAnyMarket($bean);
    $this->assertTrue($result, 'Bean should be available if ANY CrawlUrl is available');
}

public function testIsAvailableForCountry(): void
{
    // Create bean with CrawlUrl from config with market serving DE
    $bean = $this->createBeanWithMarketConfig('DE', true);

    $result = $this->repository->isAvailableForCountry($bean, 'DE');
    $this->assertTrue($result, 'Bean should be available for DE');

    $result = $this->repository->isAvailableForCountry($bean, 'FR');
    $this->assertFalse($result, 'Bean should not be available for FR');
}

public function testFindByRequestWithMarketFiltering(): void
{
    // Create test data: beans available in different markets
    $this->createBeanAvailableInMarket('DE', 'Bean DE');
    $this->createBeanAvailableInMarket('US', 'Bean US');

    // Query for available beans in DE market
    $results = $this->repository->findByRequest([
        'available' => true,
        'visitorCountryId' => 'DE',
    ]);

    $this->assertCount(1, $results);
    $this->assertEquals('Bean DE', $results[0]->getName());
}

public function testFindWithRelationshipsPreventsNPlusOneQueries(): void
{
    // Create beans with multiple CrawlUrls, Markets, AffiliatePrograms
    $beans = $this->createBeansWithComplexRelationships(10);
    $beanIds = array_map(fn($b) => $b->getId(), $beans);

    // Enable query logging
    $queryLogger = $this->enableQueryLogging();

    // Load beans with relationships
    $loadedBeans = $this->repository->findWithRelationships($beanIds);

    // Access all relationships (should not trigger additional queries)
    foreach ($loadedBeans as $bean) {
        foreach ($bean->getCrawlUrls() as $crawlUrl) {
            $config = $crawlUrl->getRoasterCrawlConfig();
            $market = $config->getMarket();
            if ($market) {
                $affiliateProgram = $market->getAffiliateProgram();
                $countries = $config->getShipsTo();
            }
        }
    }

    // Verify only one query was executed (the initial load)
    $queries = $queryLogger->getQueries();
    $this->assertLessThanOrEqual(2, count($queries), 'Should not have N+1 queries');
}

Testing Strategy

Unit Tests

1. CacheInvalidationSubscriber:
2. Test each entity type invalidates correct tags
3. Test tag uniqueness (no duplicates)

4. Test cascading invalidations

5. CoffeeBeanRepository:

6. Test isAvailableInAnyMarket() with various scenarios
7. Test isAvailableForCountry() with active/inactive markets
8. Test findByRequest() with market filtering
9. Test findWithRelationships() prevents N+1 queries

Integration Tests

1. Cache Invalidation Flow:
2. Create cached data with tags
3. Modify entity
4. Verify cache invalidated

5. Test all entity types

6. Repository Query Performance:

7. Test with large datasets
8. Verify join optimization
9. Check query count (no N+1)

🎯 Success Criteria

• CacheInvalidationSubscriber handles all multi-market entities
• Cache tags correctly invalidate when entities change
• Repository methods support market-based filtering
• isAvailableInAnyMarket() and isAvailableForCountry() work correctly
• findByRequest() updated to use CrawlUrl.available with market joins
• findWithRelationships() prevents N+1 queries
• All tests pass
• No performance regression
• Cache hit rate remains high
• Query count acceptable (no N+1 queries)

Related Files

Files to Modify:

• src/EventListener/CacheInvalidationSubscriber.php
• src/Repository/CoffeeBeanRepository.php

Files to Create:

• src/Service/Cache/CacheTags.php (optional, for constants)
• tests/EventListener/CacheInvalidationSubscriberTest.php
• tests/Repository/CoffeeBeanRepositoryTest.php (extend existing)

Files Referenced:

• src/Entity/Market.php
• src/Entity/AffiliateProgram.php
• src/Entity/BuyingOption.php
• src/Entity/RoasterCrawlConfig.php
• src/Entity/CrawlUrl.php
• src/Entity/CoffeeBean.php
• src/Service/Market/BuyingOptionService.php (uses cache tags)

Next Steps

After completing this phase: - Proceed to Phase 5 (API Integration) to update EntityToDtoMapper and endpoints - Cache and repository ready for production use - Performance optimized with proper joins and caching