How to deal with large data collections in Symfony Doctrine Entity

When we learn to use Doctrine's ORM correctly, we can have a very powerful tool that allows us to get things done in very short time.

Before continue, you may want to check on this previous posts:

• How to get Sorted results from a To Many Relation in Symfony Entity
• How to implement One to Many Relation in Symfony with Doctrine in easy steps
• How to Implement Many to Many Relation in Symfony with Doctrine in easy steps

 Lets suppose you have Customers, and each Customer has Bills associated to it:

...
    
    /**
     * @ORM\OneToMany(targetEntity="Billing", mappedBy="customers", cascade={"persist", "remove"})
     * @ORM\OrderBy({"createdDate" = "DESC"})
     */
    private $billing;
    
    /**
     *
     */
    public function __construct()
    {
        $this->billing = new ArrayCollection();
    }
...

Then you have in code:

<?php

namespace App\CustomersModule\Api\Transformers\Customers;

use App\CustomersModule\Api\Dtos\Customer;
use App\CustomersModule\Services\DtoFactoryService;
use App\CustomersModule\Entity\Customers;

class CustomerDashboardTransformer
{
    /**
     * @var Customers
     */
    private Customers $customer;
    
    /**
     * @var Customer
     */
    private Customer $apiCustomer;
    
    /**
     * @param Customers $customer
     */
    public function __construct(Customers $customer)
    {
        $this->customer = $customer;
        $this->apiCustomer = DtoFactoryService::newDto('Customers', 'Customer');
    }
    
    public function transform(): Customer
    {
        if ($this->validate()) {
            //Customer Info
            $this->apiCustomer->id = $this->customer->getId();
            $this->apiCustomer->firstName = $this->customer->getFirstName();
            $this->apiCustomer->lastName = $this->customer->getLastName();
            $this->apiCustomer->primaryEmail = $this->customer->getPrimaryEmail();
            $this->apiCustomer->phoneNumber = $this->customer->getPhoneNumber();
            
            //Customer Address details
            $country = DtoFactoryService::newDto('Customers', 'Country');
            $country->id = $this->customer->getZipCode()->getCities()->getStates()->getCountries()->getId();
            $country->countryName = $this->customer->getZipCode()->getCities()->getStates()->getCountries()->getCountryName();
            
            $state = DtoFactoryService::newDto('Customers', 'State');
            $state->id = $this->customer->getZipCode()->getCities()->getStates()->getId();
            $state->stateName = $this->customer->getZipCode()->getCities()->getStates()->getStateName();
            $state->stateShortName = $this->customer->getZipCode()->getCities()->getStates()->getStateShortName();
            $state->country = $country;
            
            $city = DtoFactoryService::newDto('Customers', 'City');
            $city->id = $this->customer->getZipCode()->getCities()->getId();
            $city->cityName = $this->customer->getZipCode()->getCities()->getCityName();
            $city->state = $state;
            
            $zipCode = DtoFactoryService::newDto('Customers', 'ZipCode');
            $zipCode->id = $this->customer->getZipCode()->getId();
            $zipCode->zipCode = $this->customer->getZipCode()->getZipCode();
            $zipCode->city = $city;
            
            $this->apiCustomer->address = $this->customer->getAddress();
            $this->apiCustomer->zipCode = $zipCode;
            
            //Customer Billing Details
            $bills = $this->customer->getBilling();//Collection
            foreach ($bills as $bill) {
                //Bill Details
                $billDto = DtoFactoryService::newDto('Customers', 'Billing');
                $billDto->id = $bill->getId();
                $billDto->salesOrderTotal = $bill->getSalesOrderTotal();
                $billDto->amountPaid = $bill->getAmountPaid();
                $billDto->createdDate = $bill->getCreatedDate()?->format('Y-m-d');
                $billDto->updatedDate = $bill->getUpdatedDate()?->format('Y-m-d');
                $billDto->deliveredDate = $bill->getDeliveredDate()?->format('Y-m-d');
                $billDto->isPaidInFull = $bill->isIsPaidInFull();
                $billDto->reference = $bill->getReference();
                
                //Sales Details
...

So far, so good, but wait, what happen if the Customer has a very large amount of Bills?

Doctrine default behavior is to lazy loading associated collections so when you get to this line:

foreach ($bills as $bill) {

 Doctrine will attempt to load All available Bills for this customer which could cause a high impact in application performance and also may overload System memory.

The best way to deal with large collections, and in my opinion, if you know in advance the collection will have more than 25 ~ 50 objects returned, its better to declare this association as Extra Lazy:

...
    
    /**
     * @ORM\OneToMany(targetEntity="Billing", mappedBy="customers", fetch="EXTRA_LAZY", cascade={"persist", "remove"})
     * @ORM\OrderBy({"createdDate" = "DESC"})
     */
    private $billing;
    
    /**
     *
     */
    public function __construct()
    {
        $this->billing = new ArrayCollection();
    }
...

 and then work getting slices of this collection and implement some sort of pagination in the Frontend to deal with this. In the PHP Backend code we may want to do this:

...
            
            //Customer Billing Details
            $bills = $this->customer->getBilling();//Collection

            $bills->slice($this->offset, $this->length);//Paginate to prevent memory 
overflow and performance degradation

            foreach ($bills as $bill) {
                //Bill Details

...

I recommend to paginate (length) in sizes not bigger than 25 ~ 50 depending in the size in Bytes of each row stored in the Table in the DB. As a rule of thumb, the less data you pull from the Db in one go, the better performance you get.

 More on this on the Doctrine Project Web Site.